Abstract:Large Language Models (LLMs) encode meanings of words in the form of distributed semantics. Distributed semantics capture common statistical patterns among language tokens (words, phrases, and sentences) from large amounts of data. LLMs perform exceedingly well across General Language Understanding Evaluation (GLUE) tasks designed to test a model's understanding of the meanings of the input tokens. However, recent studies have shown that LLMs tend to generate unintended, inconsistent, or wrong texts as outputs when processing inputs that were seen rarely during training, or inputs that are associated with diverse contexts (e.g., well-known hallucination phenomenon in language generation tasks). Crowdsourced and expert-curated knowledge graphs such as ConceptNet are designed to capture the meaning of words from a compact set of well-defined contexts. Thus LLMs may benefit from leveraging such knowledge contexts to reduce inconsistencies in outputs. We propose a novel ensemble learning method, Interpretable Ensemble Representation Learning (IERL), that systematically combines LLM and crowdsourced knowledge representations of input tokens. IERL has the distinct advantage of being interpretable by design (when was the LLM context used vs. when was the knowledge context used?) over state-of-the-art (SOTA) methods, allowing scrutiny of the inputs in conjunction with the parameters of the model, facilitating the analysis of models' inconsistent or irrelevant outputs. Although IERL is agnostic to the choice of LLM and crowdsourced knowledge, we demonstrate our approach using BERT and ConceptNet. We report improved or competitive results with IERL across GLUE tasks over current SOTA methods and significantly enhanced model interpretability.

OERL: Enhanced Representation Learning Via Open Knowledge Graphs

Representation learning with complete semantic description of knowledge graphs

IERL: Interpretable Ensemble Representation Learning -- Combining CrowdSourced Knowledge and Distributed Semantic Representations

Knowledge-guided Open Attribute Value Extraction with Reinforcement Learning

Learning Knowledge-Enhanced Contextual Language Representations for Domain Natural Language Understanding

Knowledge Graph Embedding with Diversity of Structures

Text-Enhanced Representation Learning for Knowledge Graph.

LOKE: Linked Open Knowledge Extraction for Automated Knowledge Graph Construction

Knowledge Graph Representation Learning with Relation-Guided Aggregation and Interaction

When Phrases Meet Probabilities: Enabling Open Relation Extraction with Cooperating Large Language Models

KRL_MLCCL: Multi-label Classification Based on Contrastive Learning for Knowledge Representation Learning under Open World

Text-Graph Enhanced Knowledge Graph Representation Learning.

Relation-aware Ensemble Learning for Knowledge Graph Embedding

Ernie: Enhanced Language Representation With Informative Entities

Empowering Language Models with Knowledge Graph Reasoning for Question Answering

Enriching Documents with Compact, Representative, Relevant Knowledge Graphs

Open Domain Knowledge Extraction for Knowledge Graphs

Inductively Representing Out-of-Knowledge-Graph Entities by Optimal Estimation under Translational Assumptions

KEHRL: Learning Knowledge-Enhanced Language Representations with Hierarchical Reinforcement Learning

ENT-DESC: Entity Description Generation by Exploring Knowledge Graph

Distributed Representation Learning for Knowledge Graphs with Entity Descriptions