Abstract:Scientific literature understanding is crucial for extracting targeted information and garnering insights, thereby significantly advancing scientific discovery. Despite the remarkable success of Large Language Models (LLMs), they face challenges in scientific literature understanding, primarily due to (1) a lack of scientific knowledge and (2) unfamiliarity with specialized scientific tasks. To develop an LLM specialized in scientific literature understanding, we propose a hybrid strategy that integrates continual pre-training (CPT) and supervised fine-tuning (SFT), to simultaneously infuse scientific domain knowledge and enhance instruction-following capabilities for domain-specific <a class="link-external link-http" href="http://tasks.cIn" rel="external noopener nofollow">this http URL</a> this process, we identify two key challenges: (1) constructing high-quality CPT corpora, and (2) generating diverse SFT instructions. We address these challenges through a meticulous pipeline, including PDF text extraction, parsing content error correction, quality filtering, and synthetic instruction creation. Applying this strategy, we present a suite of LLMs: SciLitLLM, specialized in scientific literature understanding. These models demonstrate promising performance on scientific literature understanding benchmarks. Our contributions are threefold: (1) We present an effective framework that integrates CPT and SFT to adapt LLMs to scientific literature understanding, which can also be easily adapted to other domains. (2) We propose an LLM-based synthesis method to generate diverse and high-quality scientific instructions, resulting in a new instruction set -- SciLitIns -- for supervised fine-tuning in less-represented scientific domains. (3) SciLitLLM achieves promising performance improvements on scientific literature understanding benchmarks.

What problem does this paper attempt to address?

The problems that this paper attempts to solve are as follows: Large - language models (LLMs) face challenges in understanding and processing scientific literature, mainly due to the lack of scientific knowledge and unfamiliarity with specific scientific tasks. To overcome these obstacles, the paper proposes a hybrid strategy combining continuous pre - training (CPT) and supervised fine - tuning (SFT) to specifically adapt to the scientific - literature - understanding tasks. Specifically, the paper aims to: 1. **Construct a high - quality continuous pre - training corpus**: Solve the format and grammar errors introduced when extracting text from PDF documents, and filter out low - quality text paragraphs, such as references and garbled text generated during the parsing process. 2. **Generate diverse scientific instructions**: Since the existing scientific - instruction data sets are scarce and costly, the paper proposes an LLM - based synthesis method to generate diverse and high - quality scientific instructions, especially for less - represented scientific fields (such as biomedicine and materials science). Through this series of measures, the paper has developed an LLM named SciLitLLM, which is specifically used for the understanding of scientific literature, and has shown encouraging performance in relevant benchmark tests.

SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding

SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis

Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding

LLMs4Synthesis: Leveraging Large Language Models for Scientific Synthesis

S3LLM: Large-Scale Scientific Software Understanding with LLMs using Source, Metadata, and Document

SciGLM: Training Scientific Language Models with Self-Reflective Instruction Annotation and Tuning

SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models

LawLLM: Law Large Language Model for the US Legal System

An Interdisciplinary Outlook on Large Language Models for Scientific Research

Structure-aware Domain Knowledge Injection for Large Language Models

DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services

LLMs-as-Instructors: Learning from Errors Toward Automating Model Improvement

TransformLLM: Adapting Large Language Models via LLM-Transformed Reading Comprehension Text

Towards Efficient Large Language Models for Scientific Text: A Review

Scientific Large Language Models: A Survey on Biological & Chemical Domains

LLM-Collaboration on Automatic Science Journalism for the General Audience

SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research

Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model