Abstract:Learning with limited data is one of the biggest problems of machine learning. Current approaches to this issue consist in learning general representations from huge amounts of data before fine-tuning the model on a small dataset of interest. While such technique, coined transfer learning, is very effective in domains such as computer vision or natural langage processing, it does not yet solve common problems of deep learning such as model interpretability or the overall need for data. This thesis explores a different answer to the problem of learning expressive models in data constrained settings: instead of relying on big datasets to learn neural networks, we will replace some modules by known functions reflecting the structure of the data. Very often, these functions will be drawn from the rich literature of kernel methods. Indeed, many kernels can reflect the underlying structure of the data, thus sparing learning parameters to some extent. Our approach falls under the hood of "inductive biases", which can be defined as hypothesis on the data at hand restricting the space of models to explore during learning. We demonstrate the effectiveness of this approach in the context of sequences, such as sentences in natural language or protein sequences, and graphs, such as molecules. We also highlight the relationship between our work and recent advances in deep learning. Additionally, we study convex machine learning models. Here, rather than proposing new models, we wonder which proportion of the samples in a dataset is really needed to learn a "good" model. More precisely, we study the problem of safe sample screening, i.e, executing simple tests to discard uninformative samples from a dataset even before fitting a machine learning model, without affecting the optimal model. Such techniques can be used to prune datasets or mine for rare samples.

On the hazards of relating representations and inductive biases

Learning abstract visual concepts via probabilistic program induction in a Language of Thought

Inference of Abstraction for a Unified Account of Symbolic Reasoning from Data

Abstract representations of events arise from mental errors in learning and memory

Inductive reasoning in minds and machines.

A Bayesian Approach to Forward and Inverse Abstract Argumentation Problems

A Bayesian Approach to Direct and Inverse Abstract Argumentation Problems

A Relational Inductive Bias for Dimensional Abstraction in Neural Networks

People infer recursive visual concepts from just a few examples

Relational Constraints On Neural Networks Reproduce Human Biases towards Abstract Geometric Regularity

A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models

Generalization, similarity, and Bayesian inference

Inference of Abstraction for a Unified Account of Reasoning and Learning

On a K-Dimensional System of Inductive Logic

On Inductive Biases for Machine Learning in Data Constrained Settings

The Compositional Structure of Bayesian Inference

Understanding Epistemic Language with a Bayesian Theory of Mind

Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning

Human-like Few-Shot Learning via Bayesian Reasoning over Natural Language

Bayes in the age of intelligent machines

Grounding Language about Belief in a Bayesian Theory-of-Mind