Machine learning at scale has led to impressive results ranging from text-based image generation, reasoning with natural language, and code synthesis to name but a few. ML at scale is also successfully applied to a broad range of problems in engineering and the sciences. These recent developments make some of us question the utility of incorporating prior knowledge in the form of symbolic (discrete) structures and algorithms. Are computing and data at scale all we need?

We will make an argument that discrete (symbolic) structures and algorithms in machine learning models are advantageous and even required in numerous application domains such as Biology, Material Science, and Physics. Biomedical entities and their structural properties, for example, can be represented as graphs and require inductive biases equivariant to certain group operations. My lab's research is concerned with the development of machine learning methods that combine discrete structures with continuous equivariant representations. We also address the problem of learning and leveraging structure from data where it is missing, combining discrete algorithms and probabilistic models with gradient-based learning. We will show that discrete structures and algorithms appear in numerous places such as ML-based PDE solvers and that modeling them explicitly is indeed beneficial. Especially machine learning models with the aim to exhibit some form of explanatory properties have to rely on symbolic representations. The talk will also cover some biomedical and physics-related applications.

Deep Bidirectional Language-Knowledge Graph Pretraining

Guaranteed Conformance of Neurosymbolic Dynamics Models to Natural Constraints

VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programming

Learning Symbolic Representations Through Joint GEnerative and DIscriminative Training

SlotDiffusion: Unsupervised Object-Centric Learning with Diffusion Models

Discovering Graph Generation Algorithms

LAMBADA: Backward Chaining for Automated Reasoning in Natural Language

Symbolic Disentangled Representations in Hyperdimensional Latent Space

A-NeSI: A Scalable Approximate Method for Probabilistic Neurosymbolic Inference

Despite recent successes, deep learning systems are still limited by their lack of generalization. I'll present an approach to addressing this limitation which combines probabilistic, model-based learning, symbolic learning and deep learning. My work centers around probabilistic programming which is a powerful abstraction layer that separates Bayesian modeling and inference. In the first part of the talk, I’ll describe “inference compilation”, an approach to amortized inference in universal probabilistic programs. In the second part of the talk, I’ll introduce a family of wake-sleep algorithms for learning model parameters. Finally, I’ll introduce a neurosymbolic generative model called “drawing out of distribution”, or DooD, which allows for out of distribution generalization for drawings.

Deep Generative Symbolic Regression with Monte-Carlo-Tree-Search

Open-Ended Dreamer: An Unsupervised Diversity-oriented Neurosymbolic Learner

Slot-VAE: Object-Centric Scene Inference and Generation with Slot Attention

Generating Temporal Logical Formulas with Transformer GANs

Using Domain Knowledge to Guide Dialog Structure Induction via Neural Probabilistic Soft Logic

Grounded physical language understanding with probabilistic programs and simulated worlds

1. Remote Poster: Deep Bidirectional Language-Knowledge Graph Pretraining
2. Remote Poster: Guaranteed Conformance of Neurosymbolic Dynamics Models to Natural Constraints
3. Remote Poster: VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programming
4. Remote Poster: Learning Symbolic Representations Through Joint GEnerative and DIscriminative Training
5. Remote Poster: SlotDiffusion: Unsupervised Object-Centric Learning with Diffusion Models
6. In-Person Poster: Discovering Graph Generation Algorithms
7. Remote Poster: LAMBADA: Backward Chaining for Automated Reasoning in Natural Language
8. In-Person Poster: Symbolic Disentangled Representations in Hyperdimensional Latent Space
9. In-Person Poster: A-NeSI: A Scalable Approximate Method for Probabilistic Neurosymbolic Inference
10. Remote Poster: Deep Generative Symbolic Regression with Monte-Carlo-Tree-Search
11. Remote Poster: Open-Ended Dreamer: An Unsupervised Diversity-oriented Neurosymbolic Learner
12. Remote Poster: Slot-VAE: Object-Centric Scene Inference and Generation with Slot Attention
13. Remote Poster: Generating Temporal Logical Formulas with Transformer GANs
14. Remote Poster: Using Domain Knowledge to Guide Dialog Structure Induction via Neural Probabilistic Soft Logic
15. Remote Poster: Grounded physical language understanding with probabilistic programs and simulated worlds

Humans display a remarkable capacity for discovering useful abstractions to make sense of and interact with the world. In particular, many of these abstractions are portable across behavioral domains, manifesting in what people see, do, and talk about. For example, people can visually decompose objects into parts; these parts can be rearranged to create new objects; the procedures for doing so can be encoded in language. What principles explain why some abstractions are favored by humans more than others, and what would it take for machines to emulate human-like learning of such “bridging” abstractions? In the first part of this talk, I’ll discuss a line of work investigating how people learn to communicate about shared procedural abstractions during collaborative physical assembly, which we formalize by combining a model of linguistic convention formation with a mechanism for inferring recurrent subroutines within the motor programs used to build various objects. In the second part, I’ll share new insights gained from extending this approach to understand why the kinds of abstractions that people learn and use varies between contexts. I will close by suggesting that embracing the study of such multimodal, naturalistic behaviors in humans at scale may shed light on the mechanisms needed to support fast, flexible learning and generalization in machines.

Many expect that AI will go from powering chatbots to providing mental health services. That it will go from advertisement to deciding who is given bail. The expectation is that AI will solve society’s problems by simply being more intelligent than we are. Implicit in this bullish perspective is the assumption that AI will naturally learn to reason from data: that it can form trains of thought that make sense, similar to how a mental health professional or judge might reason about a case, or more formally, how a mathematician might prove a theorem. This talk will investigate the question whether this behavior can be learned from data, and how we can design the next generation of AI techniques that can achieve such capabilities, focusing on constrained language generation, neuro-symbolic learning and tractable deep generative models.

Harnessing program induction, coupling robustness, and expressivity, by combining some form of symbolic and procedural knowledge appears to be a promising direction towards more open-ended innovation, and extrapolative behavior. Building upon DreamCoder framework (Ellis et al., 2021), we present an unsupervised diversity-oriented neurosymbolic learner: Open-Ended Dreamer (OED). Balancing environmental, language and novelty pressures, OED aims to learn novel, and useful programmatic abstractions. As a first test-bed we experiment with a tower building environment, where we analyze the benefits of library learning, neural guidance, innate priors, or environmental pressures to guide the formation of symbolic knowledge and open-ended program discovery.

Human language richly invokes our intuitive physical knowledge. We talk about physical objects, scenes, properties, and events; and we can make predictions and draw inferences about physical worlds described entirely in language. Understanding this everyday language requires inherently probabilistic reasoning---over possible physical worlds invoked in language and over uncertainty inherent to those physical worlds. In this paper, we propose \textbf{PiLoT}, a neurosymbolic generative model that translates language into probabilistic programs grounded in a physics engine. Our model integrates a large language-code model to robustly parse language into program expressions and uses a probabilistic physics engine to support inferences over scenes described in language. We construct a \textbf{linguistic reasoning benchmark} based on prior psychophysics experiments that requires reasoning about physical outcomes based on linguistic scene descriptions. We show that PiLoT well predicts human judgments and outperforms LLM baselines.

In safety-critical robotics and medical applications, it is common to use deep neural networks to capture the evolution of dynamical systems. This is particularly useful in modeling medical systems where data can be leveraged to individualize treatment. It is important that the data-driven model is conformant to established knowledge from the natural sciences. Such knowledge is often available or can often be distilled into a (possibly black-box) model $M$. For instance, the unicycle model (which encodes Newton's laws) for an F1 racing car. Here, we wish to best approximate the system model while being only a bounded distance away from $M$ with guarantees. We generate an unlabelled dataset to enforce conformance, where data is absent. Our first step is to distill all our data into few representative samples called memories, using the idea of a growing neural gas. Next, using these memories, we partition the state space into disjoint subsets and compute bounds for each subset utilizing $M$. This serves as a symbolic wrapper for guaranteed conformance. We argue theoretically that this only leads to bounded increase in approximation error; which can be controlled by increasing the number of memories. We experimentally show that on three case studies (Car Model, Drones, and Artificial Pancreas), our constrained neurosymbolic models conform to specified $M$ models (each encoding various constraints) with order-of-magnitude improvements compared to the augmented Lagrangian and vanilla training methods. Our code can be found at https://github.com/neurosymbolic-models/constrained_dynamics .

We provide a novel approach to construct generative models for graphs. Instead of using the traditional probabilistic models or deep generative models, we propose to instead find an algorithm that generates the data. We achieve this using evolutionary search and a powerful fitness function, implemented by a randomly initialized graph neural network. This brings certain advantages over current deep generative models, for instance, a higher potential for out-of-training-distribution generalization and direct interpretability, as the final graph generative process is expressed as a Python function. We show that this approach can be competitive with deep generative models and under some circumstances can even find the true graph generative process, and as such perfectly generalize.

Dialog Structure Induction (DSI) is the task of inferring the latent dialog structure (i.e., a set of dialog states and their temporal transitions) of a given goal-oriented dialog. It is a critical component for modern dialogue system design and discourse analysis. Existing DSI approaches are often purely data-driven, deploy models that infer latent states without access to domain knowledge, underperform when the training corpus is limited/noisy, or have difficulty when test dialogs exhibit distributional shifts from the training domain. This work explores a neural-symbolic approach as a potential solution to these problems. We introduce Neural Probabilistic Soft Logic Dialog Structure Induction (NEUPSL DSI), a principled approach that injects symbolic knowledge into the latent space of a generative neural model. Over three unsupervised dialog structure induction datasets the injection of symbolic knowledge using NEUPSL DSI provides a consistent boost in performance over the canonical baselines.

Training neural networks requires large amounts of training data, often not readily available in symbolic reasoning domains. In this extended abstract, we consider the scarcity of training data for temporal logics. We summarize a recently performed study on the capabilities of GANs and Wasserstein GANs equipped with Transformer encoders to generate sensible and challenging formulas in the prototypical temporal logic LTL. The approach produces novel and unique formula instances without the need for autoregression. The generated data can be used as substitute for real training data when training a classifier, and training data can be generated from a dataset that is too small to be trained on directly.

The idea of the disentangled representations is to reduce the data to a set of generative factors which generate it. Usually, such representations are vectors in the latent space, in which each coordinate corresponds to one of the generative factors. Then the object represented in this way can be modified by changing the value of a specific coordinate. But first, we need to determine which coordinate handles the desired generative factor, which can be complex with a high vector dimension. In this paper, we propose to represent each generative factor as a vector of the same dimension as the resulting representation. This is possible by using Hyperdimensional Computing principles (also known as Vector Symbolic Architectures), which represent symbols as high-dimensional vectors. They allow us to operate on symbols using vector operations, which leads to a simple and interpretable modification of the object in the latent space. We show it on the objects from dSprites and CLEVR datasets and provide an extensive analysis of learned symbolic disentangled representations in hyperdimensional latent space.

We introduce GEDI, a Bayesian framework that combines existing self-supervised learning objectives with likelihood-based generative models. This framework leverages the benefits of both GEnerative and DIscriminative approaches, resulting in improved symbolic representations over standalone solutions. Additionally, GEDI can be easily integrated and trained jointly with existing neuro-symbolic frameworks without the need for additional supervision or costly pre-training steps. We demonstrate through experiments on real-world data, including SVHN, CIFAR10, and CIFAR100, that GEDI outperforms existing self-supervised learning strategies in terms of clustering performance by a significant margin. The symbolic component further allows it to leverage knowledge in the form of logical constraints to improve performance in the small data regime.

Slot attention has shown remarkable object-centric representation learning performance in computer vision tasks without requiring any supervision. Despite of its object-centric binding ability brought by compositional modelling, as a deterministic module, slot attention lacks the ability to generate novel scenes. In this paper, we propose the Slot-VAE, a generative model that integrates slot attention with the hierarchical VAE framework for object-centric structured image generation. From each image, the model simultaneously infers a global scene representation to capture high-level scene structure and object-centric slot representations to embed individual object components. During generation, slot representations are generated from global scene representation to ensure coherent scene structure. Our experiments demonstrate that Slot-VAE achieves better scene structure accuracy and sample quality compared to slot-based baselines.

We present VAEL, a neuro-symbolic generative model integrating variational autoencoders (VAE) with the reasoning capabilities of probabilistic logic (L) programming. Besides standard latent subsymbolic variables, our model exploits a probabilistic logic program to define a further structured representation, which is used for logical reasoning. The entire process is end-to-end differentiable. Once trained, VAEL can solve new unseen generation tasks by (i) leveraging the previously acquired knowledge encoded in the neural component and (ii) exploiting new logical programs on the structured latent space. Our experiments provide support on the benefits of this neuro-symbolic integration both in terms of task generalization and data efficiency. To the best of our knowledge, this work is the first to propose a general-purpose end-to-end framework integrating probabilistic logic programming into a deep generative model.

We study the problem of combining neural networks with symbolic reasoning. Frameworks for Probabilistic Neurosymbolic Learning (PNL) like DeepProbLog perform exponential-time exact inference that is limited in scalability. We introduce Approximate Neurosymbolic Inference (A-NeSI): a new framework for PNL that uses deep generative modelling for scalable approximate inference. A-NeSI 1) performs approximate inference in polynomial time; 2) is trained using data generated by background knowledge; 3) can generate symbolic explanations of predictions; and 4) can guarantee the satisfaction of logical constraints at test time. Our experiments show that A-NeSI is the first end-to-end method to scale the Multi-digit MNISTAdd benchmark to sums of 15 MNIST digits.

Symbolic regression (SR) is the problem of learning a symbolic expression from numerical data. Recently, deep neural models showed competitive performance compared to more classical Genetic Programming (GP) ones. Unlike their GP counterparts, they are trained to generate expressions from datasets given as context. This allows them to produce accurate expressions in a single forward pass at test time. However, they usually do not benefit from search abilities, which result in low performance compared to GP on out-of-distribution datasets. In this paper, we propose a novel method which provides the best of both worlds, based on a Monte-Carlo Tree Search procedure using a context-aware neural mutation model, which is initially pre-trained to learn promising mutations, and further refined from successful experiences in an online fashion. The approach demonstrates state-of-the-art performance on the well-known \texttt{SRBench} benchmark.

Remarkable progress has been made on automated reasoning with natural text, by using Large Language Models (LLMs) and methods such as Chain-of-Thought prompting and Selection-Inference. These techniques search for proofs in the forward direction from axioms to the conclusion, which suffers from a combinatorial explosion of the search space, and thus high failure rates for problems requiring longer chains of reasoning. The classical automated reasoning literature has shown that reasoning in the backward direction (i.e. from the intended conclusion to supporting axioms) is significantly more efficient at proof-finding. Importing this intuition into the LM setting, we develop a neuro-symbolic Backward Chaining algorithm, called LAMBADA, that decomposes reasoning into four sub-modules, that are simply implemented by few-shot prompted LLM inference. We show that LAMBADA achieves sizable accuracy boosts over state-of-the-art forward reasoning methods on two challenging logical reasoning datasets, particularly when deep and accurate proof chains are required.

Object-centric learning aims to decompose the visual data into a set of individual entities, which is distinct from traditional deep learning models that represent a scene with a global feature. Leveraging advanced architectures such as Transformer decoders, slot-based models have shown promising results in unsupervised object discovery from naturalistic inputs. In this paper, we instead focus on the slot-to-image reconstruction quality of these models, a previously overlooked topic which is important for generation tasks such as video prediction and scene editing. Despite great segmentation outputs, recent unsupervised slot models produce blurry images and temporally inconsistent videos. We address this problem by introducing slot-conditioned diffusion models due to their strong generation capacity. Our proposed method, SlotDiffusion, not only achieves better unsupervised segmentation performance, but also generates results of higher quality compared to previous state-of-the-art on both image and video datasets.

Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks. Recent works show that a knowledge graph (KG) can complement text data, offering structured background knowledge that provides a useful scaffold for reasoning. However, these works are not pretrained to learn a deep fusion of the two modalities at scale, limiting the potential to acquire fully joint representations of text and KG. Here we propose DRAGON (Deep Bidirectional Language-Knowledge Graph Pretraining), a self-supervised approach to pretraining a deeply joint language-knowledge foundation model from text and KG at scale. Specifically, our model takes pairs of text segments and relevant KG subgraphs as input and bidirectionally fuses information from both modalities. We pretrain this model by unifying two self-supervised reasoning tasks, masked language modeling and KG link prediction. DRAGON outperforms existing LM and LM+KG models on diverse downstream tasks including question answering across general and biomedical domains, with +5% absolute gain on average. In particular, DRAGON achieves notable performance on complex reasoning about language and knowledge (+10% on questions involving long contexts or multi-step reasoning) and low-resource QA (+8% on OBQA and RiddleSense), and new state-of-the-art results on various BioNLP tasks.