In-Context Learning State Vector with Inner and Momentum Optimization

Dongfang Li,Zhenyu Liu,Xinshuo Hu,Zetian Sun,Baotian Hu,Min Zhang

2024-07-04

Abstract:Large Language Models (LLMs) have exhibited an impressive ability to perform In-Context Learning (ICL) from only a few examples. Recent works have indicated that the functions learned by ICL can be represented through compressed vectors derived from the transformer. However, the working mechanisms and optimization of these vectors are yet to be thoroughly explored. In this paper, we address this gap by presenting a comprehensive analysis of these compressed vectors, drawing parallels to the parameters trained with gradient descent, and introduce the concept of state vector. Inspired by the works on model soup and momentum-based gradient descent, we propose inner and momentum optimization methods that are applied to refine the state vector progressively as test-time adaptation. Moreover, we simulate state vector aggregation in the multiple example setting, where demonstrations comprising numerous examples are usually too lengthy for regular ICL, and further propose a divide-and-conquer aggregation method to address this challenge. We conduct extensive experiments using Llama-2 and GPT-J in both zero-shot setting and few-shot setting. The experimental results show that our optimization method effectively enhances the state vector and achieves the state-of-the-art performance on diverse tasks. Code is available at <a class="link-external link-https" href="https://github.com/HITsz-TMG/ICL-State-Vector" rel="external noopener nofollow">this https URL</a>

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address the problem of how to optimize and utilize compressed vectors to improve the performance of large language models (LLMs) in In-Context Learning (ICL). Specifically, the paper focuses on the following aspects: 1. **Mechanism and Optimization of Compressed Vectors**: - The paper explores the mechanism of compressed vectors in ICL and draws an analogy with parameters trained through gradient descent, introducing the concept of state vectors. - The authors introduce inner optimization and momentum optimization methods to gradually improve the state vectors, enhancing adaptability during testing. 2. **Challenges in Multi-Example Settings**: - When dealing with multiple examples, conventional ICL methods often encounter the problem of context length limitations. To address this, the paper proposes a divide-and-conquer aggregation method, compressing multiple examples into a single state vector, effectively solving this challenge. 3. **Experimental Validation**: - The authors conducted extensive experiments using Llama-2 and GPT-J in zero-shot and few-shot settings to validate the effectiveness of the proposed methods. The experimental results demonstrate that the optimized state vectors achieve state-of-the-art performance across various tasks. In summary, the main goal of the paper is to improve the performance of ICL in large language models by optimizing and aggregating state vectors, particularly in multi-example settings.

In-Context Learning State Vector with Inner and Momentum Optimization

Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent As Meta-Optimizers

Towards Multimodal In-Context Learning for Vision & Language Models

What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization

Vector-ICL: In-context Learning with Continuous Vector Representations

Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning

Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective

Implicit In-context Learning

Iterative Forward Tuning Boosts In-Context Learning in Language Models

In-Context Language Learning: Architectures and Algorithms

In-Context Learning Demonstration Selection via Influence Analysis

Why Larger Language Models Do In-context Learning Differently?

LIVE: Learnable In-Context Vector for Visual Question Answering

Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanism

Do pretrained Transformers Learn In-Context by Gradient Descent?

What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning

Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition

Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning

Unifying Demonstration Selection and Compression for In-Context Learning

Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Science

Decoding In-Context Learning: Neuroscience-inspired Analysis of Representations in Large Language Models