AMPO: Automatic Multi-Branched Prompt Optimization

Sheng Yang,Yurong Wu,Yan Gao,Zineng Zhou,Bin Benjamin Zhu,Xiaodi Sun,Jian-Guang Lou,Zhiming Ding,Anbang Hu,Yuan Fang,Yunsong Li,Junyan Chen,Linjun Yang

2024-10-11

Abstract:Prompt engineering is very important to enhance the performance of large language models (LLMs). When dealing with complex issues, prompt engineers tend to distill multiple patterns from examples and inject relevant solutions to optimize the prompts, achieving satisfying results. However, existing automatic prompt optimization techniques are only limited to producing single flow instructions, struggling with handling diverse patterns. In this paper, we present AMPO, an automatic prompt optimization method that can iteratively develop a multi-branched prompt using failure cases as feedback. Our goal is to explore a novel way of structuring prompts with multi-branches to better handle multiple patterns in complex tasks, for which we introduce three modules: Pattern Recognition, Branch Adjustment, and Branch Pruning. In experiments across five tasks, AMPO consistently achieves the best results. Additionally, our approach demonstrates significant optimization efficiency due to our adoption of a minimal search strategy.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in complex tasks, the existing automatic prompt optimization techniques can only generate single - flow instructions and are difficult to handle diverse patterns. Specifically, when facing complex problems, it is necessary to classify the problems so that large - language models (LLMs) can analyze the problems more systematically and find appropriate solutions. However, current methods mainly provide more details by carefully rewriting certain steps, which are still essentially single - flow instructions and are insufficient when dealing with diverse patterns.

To solve this problem, the paper proposes AMPO (Automatic Multi - Branched Prompt Optimization), an automatic prompt optimization method that can iteratively develop multi - branched prompts using failure cases as feedback. The goal of AMPO is to explore a new way of prompt structuring to better handle multiple patterns in complex tasks through multi - branching. To this end, the paper introduces three modules:

1. **Pattern Recognition**: Responsible for analyzing the patterns of failure cases.
2. **Branch Adjustment**: Adaptively selects whether to add new branches to deal with newly emerging patterns or to strengthen existing branches to provide more details. The appropriate balance is crucial for the adaptive prompt structure.
3. **Branch Pruning**: Adopts pre - pruning and post - pruning techniques to prevent prompt over - fitting and further ensure the adaptability of the prompt structure.

Through these modules, AMPO can enable the target LLMs to independently determine the best path for each sample when handling complex tasks, thereby improving their ability to handle diverse patterns and complexity. Experimental results show that AMPO consistently performs best on five tasks and, due to the adoption of the minimum search strategy, shows impressive optimization efficiency.