Abstract:Knowledge graphs (KGs) generated by large language models (LLMs) are becoming increasingly valuable for Retrieval-Augmented Generation (RAG) applications that require knowledge-intensive reasoning. However, existing KG extraction methods predominantly rely on prompt-based approaches, which are inefficient for processing large-scale corpora. These approaches often suffer from information loss, particularly with long documents, due to the lack of specialized design for KG construction. Additionally, there is a gap in evaluation datasets and methodologies for ontology-free KG construction. To overcome these limitations, we propose SynthKG, a multi-step, document-level ontology-free KG synthesis workflow based on LLMs. By fine-tuning a smaller LLM on the synthesized document-KG pairs, we streamline the multi-step process into a single-step KG generation approach called Distill-SynthKG, substantially reducing the number of LLM inference calls. Furthermore, we re-purpose existing question-answering datasets to establish KG evaluation datasets and introduce new evaluation metrics. Using KGs produced by Distill-SynthKG, we also design a novel graph-based retrieval framework for RAG. Experimental results demonstrate that Distill-SynthKG not only surpasses all baseline models in KG quality -- including models up to eight times larger -- but also consistently excels in retrieval and question-answering tasks. Our proposed graph retrieval framework also outperforms all KG-retrieval methods across multiple benchmark datasets. We release the SynthKG dataset and Distill-SynthKG model publicly to support further research and development.

Enhancing text generation from knowledge graphs with cross-structure attention distillation

Alignahead: Online Cross-Layer Knowledge Extraction on Graph Neural Networks

Online Cross-Layer Knowledge Distillation on Graph Neural Networks with Deep Supervision

Generating Faithful Text From a Knowledge Graph with Noisy Reference Text

Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph Completion

Contextualization Distillation from Large Language Model for Knowledge Graph Completion

Distill-SynthKG: Distilling Knowledge Graph Synthesis Workflow for Improved Coverage and Efficiency

Cross-domain knowledge distillation for text classification

GenDistiller: Distilling Pre-trained Language Models based on Generative Models

Multi-Granularity Structural Knowledge Distillation for Language Model Compression

Bi-Directional Multi-Granularity Generation Framework for Knowledge Graph-to-Text with Large Language Model

Online Adversarial Distillation for Graph Neural Networks

Unlock the Power: Competitive Distillation for Multi-Modal Large Language Models

JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs

ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling

Pre-training Distillation for Large Language Models: A Design Space Exploration

Exploring Graph-based Knowledge: Multi-Level Feature Distillation via Channels Relational Graph

Tree-structured Auxiliary Online Knowledge Distillation

Unveiling the Magic: Investigating Attention Distillation in Retrieval-augmented Generation

Enhanced Scalable Graph Neural Network via Knowledge Distillation