Abstract:Reasoning, the ability to logically draw conclusions from existing knowledge, is a hallmark of human. Together with perception, they constitute the two major themes of artificial intelligence. While deep learning has pushed the limit of perception beyond human-level performance, the progress in reasoning domains is way behind. One fundamental reason is that reasoning problems usually have flexible structures for both knowledge and queries, and many existing models only perform well on structures seen during training. Here we aim to push the boundary of reasoning models by devising algorithms that generalize across knowledge and query structures, as well as systems that accelerate development on structured data. This thesis consists of three parts. In Part I, we study models that can inductively generalize to unseen knowledge graphs with new entity and relation vocabularies. For new entities, we propose a framework that learns neural operators in a dynamic programming algorithm computing path representations. For relations, we construct a relation graph to capture the interactions between relations, thereby converting new relations into new entities. In Part II, we propose two solutions for generalizing across multi-step queries on knowledge graphs and text respectively. For knowledge graphs, we show that multi-step queries can be solved by multiple calls of graph neural networks and fuzzy logic operations. For text, we devise an algorithm to learn explicit knowledge as textual rules to improve large language models on multi-step queries. In Part III, we propose two systems to facilitate machine learning development on structured data. Our library treats structured data as first-class citizens and removes the barrier for developing algorithms on structured data. Our node embedding system solves the GPU memory bottleneck of embedding matrices and scales to graphs with billion nodes.

Multi-Step Deductive Reasoning Over Natural Language: An Empirical Study on Out-of-Distribution Generalisation

Concise and Organized Perception Facilitates Large Language Models for Deductive Reasoning.

Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models

Boosting Deductive Reasoning with Step Signals In RLHF

Assessing and Enhancing the Robustness of Large Language Models with Task Structure Variations for Logical Reasoning

Abstract Meaning Representation-Based Logic-Driven Data Augmentation for Logical Reasoning

Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

Scaling Synthetic Logical Reasoning Datasets with Context-Sensitive Declarative Grammars

ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning

DeepLogic: Joint Learning of Neural Perception and Logical Reasoning.

Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Models

Certified Deductive Reasoning with Language Models

Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models

LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models

Multi-step Inference over Unstructured Data

Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach

Learning Representations for Reasoning: Generalizing Across Diverse Structures

Reliable Reasoning Beyond Natural Language

Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Data

Out-of-Distribution Generalization by Neural-Symbolic Joint Training.