Abstract:Large language models (LLMs) are becoming increasingly important for machine learning applications. However, it can be challenging to align LLMs with our intent, particularly when we want to generate content that is preferable over others or when we want the LLM to respond in a certain style or tone that is hard to describe. To address this challenge, we propose an approach that uses contrastive examples to better describe our intent. This involves providing positive examples that illustrate the true intent, along with negative examples that show what characteristics we want LLMs to avoid. The negative examples can be retrieved from labeled data, written by a human, or generated by the LLM itself. Before generating an answer, we ask the model to analyze the examples to teach itself what to avoid. This reasoning step provides the model with the appropriate articulation of the user's need and guides it towards generting a better answer. We tested our approach on both synthesized and real-world datasets, including StackExchange and Reddit, and found that it significantly improves performance compared to standard few-shot prompting

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address the issue of large language models (LLMs) struggling to align with user intent when generating content. Specifically, when there is a need to generate content in a specific style or tone, existing LLMs often fail to meet user requirements. To tackle this challenge, the authors propose a method based on contrastive examples (Contrastive In-Context Learning), which uses positive and negative examples to better describe user intent. ### Main Contributions 1. **Proposed a new method**: Utilizing contrastive examples (including both positive and negative examples) to improve the quality of content generated by LLMs. 2. **Experimental validation**: Conducted experiments on synthetic datasets and real-world datasets (such as StackExchange and Reddit), showing that this method significantly improves performance. 3. **Demonstrated the potential of negative examples**: Negative examples can more effectively guide LLMs to generate content that aligns with user preferences. ### Method Overview 1. **Obtaining contrastive examples**: - **Using annotated feedback**: Extract high-scoring and low-scoring responses from user feedback as positive and negative examples. - **Automatically generating negative examples**: If annotated feedback is unavailable, LLM can generate responses to be used as negative examples. - **Automated evaluation**: For certain tasks, automatic evaluation rules can be defined to select positive and negative examples. 2. **Forming prompts**: - **Contrastive examples as few-shot examples**: Provide contrastive example pairs as few-shot examples to the LLM. - **Inference and analysis**: Require the LLM to analyze the characteristics of positive and negative examples, then generate a response that aligns with user preferences. ### Experimental Setup - **Datasets**: Including StackExchange, Reddit, and synthetic datasets. - **Models**: Used non-dialogue LLMs (such as GPT-3) and dialogue LLMs (such as ChatGPT and GPT-4). - **Evaluation methods**: Including reference-based evaluations (such as BERT Score and sentence embedding similarity) and reference-free evaluations (such as DialogRPT and GPT Score). ### Experimental Results - **Contrastive-Combined method**: Combining contrastive examples with instructions performed best in most cases. - **Source of negative examples**: Both human-written and LLM-generated negative examples effectively improved performance. - **Reducing prompt length**: Compressing contrastive examples into brief instructions can reduce prompt length and cost. ### Conclusion By introducing contrastive examples, especially negative examples, the quality of content generated by LLMs can be significantly improved to better align with user preferences. This method not only performs well on synthetic datasets but also achieves significant improvements on real-world datasets.

Customizing Language Model Responses with Contrastive In-Context Learning

Self-Instructed Derived Prompt Generation Meets In-Context Learning: Unlocking New Potential of Black-Box LLMs

Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding

Leveraging Large Language Models for Multiple Choice Question Answering

Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions

I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responses

Evolutionary Contrastive Distillation for Language Model Alignment

Multimodal Contrastive In-Context Learning

Enhancing Large Language Model Performance To Answer Questions and Extract Information More Accurately

The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting

Supervised Knowledge Makes Large Language Models Better In-context Learners

Personalized LLM for Generating Customized Responses to the Same Query from Different Users

CELL your Model: Contrastive Explanations for Large Language Models

Interpreting Language Reward Models via Contrastive Explanations

Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning

Large Language Models Know What Makes Exemplary Contexts

Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning

Enhancing Dialogue Generation Via Multi-Level Contrastive Learning.

Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learning

Learning from Contrastive Prompts: Automated Optimization and Adaptation

Respond in my Language: Mitigating Language Inconsistency in Response Generation based on Large Language Models