Abstract:The commonsense natural language inference (CNLI) tasks aim to select the most likely follow-up statement to a contextual description of ordinary, everyday events and facts. Current approaches to transfer learning of CNLI models across tasks require many labeled data from the new task. This paper presents a way to reduce this need for additional annotated training data from the new task by leveraging symbolic knowledge bases, such as ConceptNet. We formulate a teacher-student framework for mixed symbolic-neural reasoning, with the large-scale symbolic knowledge base serving as the teacher and a trained CNLI model as the student. This hybrid distillation process involves two steps. The first step is a symbolic reasoning process. Given a collection of unlabeled data, we use an abductive reasoning framework based on Grenander's pattern theory to create weakly labeled data. Pattern theory is an energy-based graphical probabilistic framework for reasoning among random variables with varying dependency structures. In the second step, the weakly labeled data, along with a fraction of the labeled data, is used to transfer-learn the CNLI model into the new task. The goal is to reduce the fraction of labeled data required. We demonstrate the efficacy of our approach by using three publicly available datasets (OpenBookQA, SWAG, and HellaSWAG) and evaluating three CNLI models (BERT, LSTM, and ESIM) that represent different tasks. We show that, on average, we achieve 63% of the top performance of a fully supervised BERT model with no labeled data. With only 1,000 labeled samples, we can improve this performance to 72%. Interestingly, without training, the teacher mechanism itself has significant inference power. The pattern theory framework achieves 32.7% accuracy on OpenBookQA, outperforming transformer-based models such as GPT (26.6%), GPT-2 (30.2%), and BERT (27.1%) by a significant margin. We demonstrate that the framework can be generalized to successfully train neural CNLI models using knowledge distillation under unsupervised and semi-supervised learning settings. Our results show that it outperforms all unsupervised and weakly supervised baselines and some early supervised approaches, while offering competitive performance with fully supervised baselines. Additionally, we show that the abductive learning framework can be adapted for other downstream tasks, such as unsupervised semantic textual similarity, unsupervised sentiment classification, and zero-shot text classification, without significant modification to the framework. Finally, user studies show that the generated interpretations enhance its explainability by providing key insights into its reasoning mechanism.

DomiKnowS: A Library for Integration of Symbolic Domain Knowledge in Deep Learning

Prompt2DeModel: Declarative Neuro-Symbolic Modeling with Natural Language

DomainLab: A modular Python package for domain generalization in deep learning

SymbolicAI: A framework for logic-based approaches combining generative models and solvers

Parallel Neurosymbolic Integration with Concordia

Symbolic Techniques for Deep Learning: Challenges and Opportunities

Diversity Boosted Learning for Domain Generalization with Large Number of Domains

Mapping the Neuro-Symbolic AI Landscape by Architectures: A Handbook on Augmenting Deep Learning Through Symbolic Reasoning

$\textit{SKIntern}$: Internalizing Symbolic Knowledge for Distilling Better CoT Capabilities into Small Language Models

DeepOnto: A Python Package for Ontology Engineering with Deep Learning

Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery

Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Models

Leveraging Symbolic Knowledge Bases for Commonsense Natural Language Inference Using Pattern Theory

A Neural-Guided Dynamic Symbolic Network for Exploring Mathematical Expressions from Data

Exploring knowledge graph-based neural-symbolic system from application perspective

Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning

Unsupervised Neural-Symbolic Integration

Knowledge Enhanced Neural Networks for relational domains

Customized Information and Domain-centric Knowledge Graph Construction with Large Language Models

Graph Enabled Cross-Domain Knowledge Transfer

Dynamic Fusion Network for Multi-Domain End-to-end Task-Oriented Dialog