Abstract:Advocates for Neuro-Symbolic Artificial Intelligence (NeSy) assert that combining deep learning with symbolic reasoning will lead to stronger AI than either paradigm on its own. As successful as deep learning has been, it is generally accepted that even our best deep learning systems are not very good at abstract reasoning. And since reasoning is inextricably linked to language, it makes intuitive sense that Natural Language Processing (NLP), would be a particularly well-suited candidate for NeSy. We conduct a structured review of studies implementing NeSy for NLP, with the aim of answering the question of whether NeSy is indeed meeting its promises: reasoning, out-of-distribution generalization, interpretability, learning and reasoning from small data, and transferability to new domains. We examine the impact of knowledge representation, such as rules and semantic networks, language structure and relational structure, and whether implicit or explicit reasoning contributes to higher promise scores. We find that systems where logic is compiled into the neural network lead to the most NeSy goals being satisfied, while other factors such as knowledge representation, or type of neural architecture do not exhibit a clear correlation with goals being met. We find many discrepancies in how reasoning is defined, specifically in relation to human level reasoning, which impact decisions about model architectures and drive conclusions which are not always consistent across studies. Hence we advocate for a more methodical approach to the application of theories of human reasoning as well as the development of appropriate benchmarks, which we hope can lead to a better understanding of progress in the field. We make our data and code available on github for further analysis.11 https://github.com/kyleiwaniec/neuro-symbolic-ai-systematic-review

System 2 Reasoning Capabilities Are Nigh

System-2 Reasoning via Generality and Adaptation

Studying and improving reasoning in humans and machines

Towards Neural Network-based Reasoning

Computational principles of intelligence: learning and reasoning with neural networks

Tunneling Neural Perception and Logic Reasoning through Abductive Learning

AI Reasoning Systems: PAC and Applied Methods

Uncovering the Data-Related Limits of Human Reasoning Research: An Analysis based on Recommender Systems

Case Study: Testing Model Capabilities in Some Reasoning Tasks

Reasoning Capacity in Multi-Agent Systems: Limitations, Challenges and Human-Centered Solutions

Neural networks for abstraction and reasoning: Towards broad generalization in machines

Neuro-Symbolic Forward Reasoning

A Feature-based Generalizable Prediction Model for Both Perceptual and Abstract Reasoning

Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Review

Is neuro-symbolic AI meeting its promises in natural language processing? A structured review

Can Large Language Models Reason? A Characterization via 3-SAT

Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge

From Recognition to Cognition: Visual Commonsense Reasoning

A neural network model of perception and reasoning

Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification

Reasoning with Large Language Models, a Survey