Abstract:This paper introduces a pioneering methodology, termed StructTuning, to efficiently transform foundation Large Language Models (LLMs) into domain specialists. It significantly reduces the training corpus requirement to a mere 0.3%, while achieving an impressive 50% of traditional knowledge injection performance. Our method is inspired by the educational processes of human students, particularly how structured domain knowledge from textbooks is assimilated and subsequently applied to tackle real-world challenges through specific exercises. Based on this, we propose a novel two-stage strategy for knowledge injection and alignment: Structure-aware Continual Pre-Training (SCPT) and Structure-aware Supervised Fine-Tuning (SSFT). In the SCPT phase, we automatically extract the domain knowledge taxonomy and reorganize the training corpora, enabling LLMs to effectively link textual segments to targeted knowledge points within the taxonomy. In the SSFT phase, we explicitly prompt models to elucidate the underlying knowledge structure in their outputs, leveraging the structured domain insight to address practical problems. Our ultimate method has undergone extensive evaluations across model architectures and scales, using closed-book question-answering tasks on LongBench and MMedBench datasets. Remarkably, our method demonstrates the potential of comparable improvement against the state-of-the-art MMedLM2 on MMedBench, while significantly reducing the training costs to 5%. This breakthrough paves the way for scaling up our StructTuning for stronger domain-specific LLMs with comprehensive data utilization. Code is available at <a class="link-external link-https" href="https://github.com/alibaba/struxgpt" rel="external noopener nofollow">this https URL</a>.

Chain of Knowledge: A Framework for Grounding Large Language Models with Structured Knowledge Bases

Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources

Chain-of-Knowledge: Integrating Knowledge Reasoning into Large Language Models by Learning from Knowledge Graphs

Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs

Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answering

Chain of Thought Prompting Elicits Knowledge Augmentation

Large Language Models and Knowledge Graphs: Opportunities and Challenges

Reasoning on Efficient Knowledge Paths:Knowledge Graph Guides Large Language Model for Domain Question Answering

CoQ:AN Empirical Framework for Multi-hop Question Answering Empowered by Large Language Models

CodeKGC: Code Language Model for Generative Knowledge Graph Construction

Chain-of-Thoughts for Molecular Understanding

CogMG: Collaborative Augmentation Between Large Language Model and Knowledge Graph

Making Large Language Models Perform Better in Knowledge Graph Completion

Large Knowledge Model: Perspectives and Challenges

SKILL: Structured Knowledge Infusion for Large Language Models

Towards Faithful Chain-of-Thought: Large Language Models are Bridging Reasoners

KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion

Structure-aware Domain Knowledge Injection for Large Language Models

ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting