Abstract:Large Language Models (LLMs) excel in data synthesis but can be inaccurate in domain-specific tasks, which retrieval-augmented generation (RAG) systems address by leveraging user-provided data. However, RAGs require optimization in both retrieval and generation stages, which can affect output quality. In this paper, we present LLM-Ref, a writing assistant tool that aids researchers in writing articles from multiple source documents with enhanced reference synthesis and handling capabilities. Unlike traditional RAG systems that use chunking and indexing, our tool retrieves and generates content directly from text paragraphs. This method facilitates direct reference extraction from the generated outputs, a feature unique to our tool. Additionally, our tool employs iterative response generation, effectively managing lengthy contexts within the language model's constraints. Compared to baseline RAG-based systems, our approach achieves a $3.25\times$ to $6.26\times$ increase in Ragas score, a comprehensive metric that provides a holistic view of a RAG system's ability to produce accurate, relevant, and contextually appropriate responses. This improvement shows our method enhances the accuracy and contextual relevance of writing assistance tools.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the deficiencies in existing writing - aid tools of Retrieval - Augmented Generation (RAG) systems when dealing with multi - source documents, especially the limitations of these systems in citation extraction and context management. Specifically: 1. **Limitations in Citation Processing**: When dealing with multi - source documents, existing RAG systems usually divide the documents into chunks, which not only affects the accurate extraction of citations but also makes it difficult for the system to provide a comprehensive list of reference documents, including primary citations (i.e., source documents) and secondary citations (i.e., other documents mentioned in the source documents). 2. **Challenges in Context Management**: Due to the limitations of the input context length of Language Model (LLM), existing RAG systems may ignore important context details when dealing with long - form content, resulting in a decline in the accuracy of the generated content. 3. **Improvement of Content Relevance**: In the retrieval stage, existing RAG systems may not be able to accurately find the paragraphs most relevant to the query, thus affecting the relevance and accuracy of the final generated content. To overcome these problems, the paper proposes a writing - aid tool named LLM - Ref. This tool achieves more accurate citation extraction and better context management by directly retrieving and generating content from text paragraphs instead of using traditional chunking and indexing methods. In addition, LLM - Ref also adopts an iterative generation method to deal with long - form context, ensuring that the generated content is both accurate and relevant. Through these improvements, LLM - Ref significantly outperforms existing RAG systems on multiple evaluation metrics, especially in multi - source document scenarios.

LLM-Ref: Enhancing Reference Handling in Technical Writing with Large Language Models

Ground Every Sentence: Improving Retrieval-Augmented LLMs with Interleaved Reference-Claim Generation

Refiner: Restructure Retrieval Content Efficiently to Advance Question-Answering Capabilities

LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback

RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation

Refiner: Restructure Retrieved Content Efficiently to Advance Question-Answering Capabilities

Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

Fine-grained LLM Agent: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback

RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation

From RAGs to riches: Using large language models to write documents for clinical trials

AssistRAG: Boosting the Potential of Large Language Models with an Intelligent Information Assistant

Meta Knowledge for Retrieval Augmented Large Language Models

SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains

Improving Retrieval for RAG based Question Answering Models on Financial Documents

On the Capacity of Citation Generation by Large Language Models

Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models

A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models

Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models

RevisEval: Improving LLM-as-a-Judge via Response-Adapted References

Effective Large Language Model Adaptation for Improved Grounding and Citation Generation

WeKnow-RAG: An Adaptive Approach for Retrieval-Augmented Generation Integrating Web Search and Knowledge Graphs