Abstract:The commonsense natural language inference (CNLI) tasks aim to select the most likely follow-up statement to a contextual description of ordinary, everyday events and facts. Current approaches to transfer learning of CNLI models across tasks require many labeled data from the new task. This paper presents a way to reduce this need for additional annotated training data from the new task by leveraging symbolic knowledge bases, such as ConceptNet. We formulate a teacher-student framework for mixed symbolic-neural reasoning, with the large-scale symbolic knowledge base serving as the teacher and a trained CNLI model as the student. This hybrid distillation process involves two steps. The first step is a symbolic reasoning process. Given a collection of unlabeled data, we use an abductive reasoning framework based on Grenander's pattern theory to create weakly labeled data. Pattern theory is an energy-based graphical probabilistic framework for reasoning among random variables with varying dependency structures. In the second step, the weakly labeled data, along with a fraction of the labeled data, is used to transfer-learn the CNLI model into the new task. The goal is to reduce the fraction of labeled data required. We demonstrate the efficacy of our approach by using three publicly available datasets (OpenBookQA, SWAG, and HellaSWAG) and evaluating three CNLI models (BERT, LSTM, and ESIM) that represent different tasks. We show that, on average, we achieve 63% of the top performance of a fully supervised BERT model with no labeled data. With only 1,000 labeled samples, we can improve this performance to 72%. Interestingly, without training, the teacher mechanism itself has significant inference power. The pattern theory framework achieves 32.7% accuracy on OpenBookQA, outperforming transformer-based models such as GPT (26.6%), GPT-2 (30.2%), and BERT (27.1%) by a significant margin. We demonstrate that the framework can be generalized to successfully train neural CNLI models using knowledge distillation under unsupervised and semi-supervised learning settings. Our results show that it outperforms all unsupervised and weakly supervised baselines and some early supervised approaches, while offering competitive performance with fully supervised baselines. Additionally, we show that the abductive learning framework can be adapted for other downstream tasks, such as unsupervised semantic textual similarity, unsupervised sentiment classification, and zero-shot text classification, without significant modification to the framework. Finally, user studies show that the generated interpretations enhance its explainability by providing key insights into its reasoning mechanism.

Knowledge Adaptive Neural Network for Natural Language Inference.

Neural Natural Language Inference Models Enhanced with External Knowledge

Natural Language Inference with External Knowledge.

Dynamic Knowledge Integration for Natural Language Inference

KGAnet: a Knowledge Graph Attention Network for Enhancing Natural Language Inference

Natural Language Inference Using Lstm Model With Sentence Fusion

Generating Knowledge Aware Explanation for Natural Language Inference

Knowledge as a Teacher: Knowledge-Guided Structural Attention Networks

Context-Aware Dual-Attention Network for Natural Language Inference

Natural Language-centered Inference Network for Multi-modal Fake News Detection

Building a commonsense knowledge base for context-awareness inference

Unsupervised Pre-training with Structured Knowledge for Improving Natural Language Inference

Asynchronous Deep Interaction Network for Natural Language Inference.

Multilingual Knowledge Graph Embeddings with Neural Networks.

Numerical Markov Logic Network: A Scalable Probabilistic Framework for Hybrid Knowledge Inference

Enhanced Lstm For Natural Language Inference

Leveraging Symbolic Knowledge Bases for Commonsense Natural Language Inference Using Pattern Theory

Semantics-Aware Inferential Network for Natural Language Understanding

Research on Attention Memory Networks As a Model for Learning Natural Language Inference.

Attention-Fused Deep Matching Network for Natural Language Inference

Multilevel Image-Enhanced Sentence Representation Net for Natural Language Inference