Abstract:Large-scale generative language and vision-language models (LLMs and VLMs) excel in few-shot in-context learning for decision making and instruction following. However, they require high-quality exemplar demonstrations in their context window. In this work, we ask: Can LLMs and VLMs generate their own examples from generic, sub-optimal demonstrations? We propose In-Context Abstraction Learning (ICAL), a method that builds a memory of multimodal experience from sub-optimal demonstrations and human feedback. Given a task demonstration that may contain inefficiencies or mistakes, a VLM abstracts the trajectory into a generalized program of thoughts by correcting inefficient actions and annotating cognitive abstractions: causal relationships, object state changes, temporal subgoals, and task-relevant visual elements. These programs of thought are iteratively improved through human feedback while the agent executes the trajectory in a similar environment. The resulting examples significantly improve decision-making in retrieval-augmented LLM and VLM agents. Moreover, as the agent's library of examples grows, it becomes more efficient, relying less on human feedback and requiring fewer environment interactions per demonstration. Our ICAL agent surpasses the SOTA in dialogue-based instruction following in TEACh, multimodal web agents in VisualWebArena, and action anticipation in Ego4D. In TEACh, we achieve a 12.6% improvement in goal-condition success. In VisualWebArena, our task success rate improves over few-shot GPT4V. In Ego4D action forecasting, we improve over few-shot GPT-4V and remain competitive with supervised models. We show finetuning our retrieval-augmented in-context agent yields additional improvements. Our approach significantly reduces reliance on manual prompt engineering and consistently outperforms in-context learning from action plans that lack such programs of thought.

Robust Memory Augmentation by Constrained Latent Imagination

Model-Based Reinforcement Learning Via Imagination with Derived Memory.

Dual Memory Model for Experience-Once Task-Incremental Lifelong Learning.

Learning a World Model With Multitimescale Memory Augmentation

Continual Learning Through Retrieval and Imagination.

Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning

Interactive Continual Learning: Fast and Slow Thinking

Memory-Augmented Image Captioning

Image Augmentation Based Momentum Memory Intrinsic Reward for Sparse Reward Visual Scenes

VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought

Efficient Intrinsically Motivated Robotic Grasping with Learning-Adaptive Imagination in Latent Space

Empowering Working Memory for Large Language Model Agents

AdaMemento: Adaptive Memory-Assisted Policy Optimization for Reinforcement Learning

Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models

$\text{Memory}^3$: Language Modeling with Explicit Memory

Concept learning through deep reinforcement learning with memory-augmented neural networks

MemoryBank: Enhancing Large Language Models with Long-Term Memory

Encode-Store-Retrieve: Augmenting Human Memory through Language-Encoded Egocentric Perception

Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes

Augmented Replay Memory in Reinforcement Learning With Continuous Control

Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memory