Abstract:Large language models are powerful text processors and reasoners, but are still subject to limitations including outdated knowledge and hallucinations, which necessitates connecting them to the world. Retrieval-augmented large language models have raised extensive attention for grounding model generation on external knowledge. However, retrievers struggle to capture relevance, especially for queries with complex information needs. Recent work has proposed to improve relevance modeling by having large language models actively involved in retrieval, i.e., to improve retrieval with generation. In this paper, we show that strong performance can be achieved by a method we call Iter-RetGen, which synergizes retrieval and generation in an iterative manner. A model output shows what might be needed to finish a task, and thus provides an informative context for retrieving more relevant knowledge which in turn helps generate a better output in the next iteration. Compared with recent work which interleaves retrieval with generation when producing an output, Iter-RetGen processes all retrieved knowledge as a whole and largely preserves the flexibility in generation without structural constraints. We evaluate Iter-RetGen on multi-hop question answering, fact verification, and commonsense reasoning, and show that it can flexibly leverage parametric knowledge and non-parametric knowledge, and is superior to or competitive with state-of-the-art retrieval-augmented baselines while causing fewer overheads of retrieval and generation. We can further improve performance via generation-augmented retrieval adaptation.

Reimagining Retrieval Augmented Language Models for Answering Queries

Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy

Query Rewriting in Retrieval-Augmented Large Language Models.

Reliable, Adaptable, and Attributable Language Models with Retrieval

More Room for Language: Investigating the Effect of Retrieval on Language Models

Retrieval-enhanced Knowledge Editing in Language Models for Multi-Hop Question Answering

Query Rewriting for Retrieval-Augmented Large Language Models

REALM: Retrieval-Augmented Language Model Pre-Training

Understanding Retrieval Augmentation for Long-Form Question Answering

Retrieval Augmented Generation for Domain-specific Question Answering

Assessing "Implicit" Retrieval Robustness of Large Language Models

Enhancing Retrieval Processes for Language Generation with Augmented Queries

Retrieval-Augmented Retrieval: Large Language Models Are Strong Zero-Shot Retriever.

Redefining Information Retrieval of Structured Database via Large Language Models

Query Optimization for Parametric Knowledge Refinement in Retrieval-Augmented Large Language Models

Improving Retrieval-Augmented Large Language Models via Data Importance Learning

RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing

Making Retrieval-Augmented Language Models Robust to Irrelevant Context

Retrieval-Generation Synergy Augmented Large Language Models

Retrieval-Augmented Generation for Large Language Models: A Survey