MDCure: A Scalable Pipeline for Multi-Document Instruction-Following

Gabrielle Kaili-May Liu,Bowen Shi,Avi Caciularu,Idan Szpektor,Arman Cohan

2024-10-31

Abstract:Multi-document (MD) processing is crucial for LLMs to handle real-world tasks such as summarization and question-answering across large sets of documents. While LLMs have improved at processing long inputs, MD contexts still present challenges, such as managing inter-document dependencies, redundancy, and incoherent structures. We introduce MDCure, a scalable and effective fine-tuning pipeline to enhance the MD capabilities of LLMs without the computational cost of pre-training or reliance on human annotated data. MDCure is based on generation of high-quality synthetic MD instruction data from sets of related articles via targeted prompts. We further introduce MDCureRM, a multi-objective reward model which filters generated data based on their training utility for MD settings. With MDCure, we fine-tune a variety of LLMs, from the FlanT5, Qwen2, and LLAMA3.1 model families, up to 70B parameters in size. Extensive evaluations on a wide range of MD and long-context benchmarks spanning various tasks show MDCure consistently improves performance over pre-trained baselines and over corresponding base models by up to 75.5%. Our code, datasets, and models are available at <a class="link-external link-https" href="https://github.com/yale-nlp/MDCure" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenges in multi - document (MD) processing, especially for large - language models (LLMs). Although existing LLMs have made improvements in handling long - input, they still face problems such as managing inter - document dependencies, redundancy, and incoherent structures in multi - document processing. These problems are particularly prominent when the documents are long. The paper proposes a scalable fine - tuning pipeline named MDCure, which aims to enhance the multi - document processing capabilities of LLMs by generating high - quality synthetic multi - document instruction data without pre - training or relying on manually - annotated data.

Specifically, the paper mainly addresses the following issues:
1. **Challenges in multi - document processing**: Managing inter - document dependencies, redundancy, contradictions, omissions, and the lack of a coherent inter - document structure.
2. **Data scarcity and complexity**: Generating high - quality multi - document instruction data to overcome data scarcity and complexity problems.
3. **Long - context processing**: Improving the performance of LLMs when handling long - context documents.
4. **Scalability and cost - effectiveness**: Providing a scalable and cost - effective method to adapt to models of different scales and types.

Through these methods, MDCure can significantly improve the performance of LLMs in a variety of multi - document benchmarks, with an average improvement of up to 75.5%.