Reasoning Graph Enhanced Exemplars Retrieval for In-Context Learning

Yukang Lin,Bingchen Zhong,Shuoran Jiang,Joanna Siebert,Qingcai Chen

2024-09-17

Abstract:Large language models(LLMs) have exhibited remarkable few-shot learning capabilities and unified the paradigm of NLP tasks through the in-context learning(ICL) technique. Despite the success of ICL, the quality of the exemplar demonstrations can significantly influence the LLM's performance. Existing exemplar selection methods mainly focus on the semantic similarity between queries and candidate exemplars. On the other hand, the logical connections between reasoning steps can be beneficial to depict the problem-solving process as well. In this paper, we proposes a novel method named Reasoning Graph-enhanced Exemplar Retrieval(RGER). RGER first quires LLM to generate an initial response, then expresses intermediate problem-solving steps to a graph structure. After that, it employs graph kernel to select exemplars with semantic and structural similarity. Extensive experiments demonstrate the structural relationship is helpful to the alignment of queries and candidate exemplars. The efficacy of RGER on math and logit reasoning tasks showcases its superiority over state-of-the-art retrieval-based approaches. Our code is released at <a class="link-external link-https" href="https://github.com/Yukang-Lin/RGER" rel="external noopener nofollow">this https URL</a>.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to select exemplars more effectively in In - Context Learning (ICL). Although large language models (LLMs) perform well in ICL, their performance highly depends on the quality of the selected exemplars. Existing exemplar selection methods mainly focus on the semantic similarity between the query and candidate exemplars, while ignoring the logical connections between reasoning steps, and the latter is also important for depicting the problem - solving process. Therefore, the paper proposes a new method - Reasoning Graph Enhanced Exemplar Retrieval (RGER), aiming to improve the effectiveness of exemplar selection by combining semantic and structural similarities. Specifically, RGER first requires the LLM to generate an initial response, and then expresses the intermediate reasoning steps as a graph structure. Next, a graph kernel is used to select exemplars with semantic and structural similarities. Experimental results show that structural relationships help align the query with candidate exemplars, and on mathematical and logical reasoning tasks, RGER outperforms existing retrieval - based methods.

Reasoning Graph Enhanced Exemplars Retrieval for In-Context Learning

GraphIC: A Graph-Based In-Context Example Retrieval Model for Multi-Step Reasoning

An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaboration

Retrieval Meets Reasoning: Dynamic In-Context Editing for Long-Text Understanding

Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation

Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning

Graphical Reasoning: LLM-based Semi-Open Relation Extraction

Graph Neural Network Enhanced Retrieval for Question Answering of LLMs

Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs

Graph Elicitation for Guiding Multi-Step Reasoning in Large Language Models

Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation

EXPLORA: Efficient Exemplar Subset Selection for Complex Reasoning

GraphLLM: Boosting Graph Reasoning Ability of Large Language Model

GIVE: Structured Reasoning with Knowledge Graph Inspired Veracity Extrapolation

GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning

Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models

RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement

Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Models

GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approach

Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks