Towards Hierarchical Multi-Agent Workflows for Zero-Shot Prompt Optimization

Yuchi Liu,Jaskirat Singh,Gaowen Liu,Ali Payani,Liang Zheng

2024-05-31

Abstract:Large language models (LLMs) have shown great progress in responding to user questions, allowing for a multitude of diverse applications. Yet, the quality of LLM outputs heavily depends on the prompt design, where a good prompt might enable the LLM to answer a very challenging question correctly. Therefore, recent works have developed many strategies for improving the prompt, including both manual crafting and in-domain optimization. However, their efficacy in unrestricted scenarios remains questionable, as the former depends on human design for specific questions and the latter usually generalizes poorly to unseen scenarios. To address these problems, we give LLMs the freedom to design the best prompts according to themselves. Specifically, we include a hierarchy of LLMs, first constructing a prompt with precise instructions and accurate wording in a hierarchical manner, and then using this prompt to generate the final answer to the user query. We term this pipeline Hierarchical Multi-Agent Workflow, or HMAW. In contrast with prior works, HMAW imposes no human restriction and requires no training, and is completely task-agnostic while capable of adjusting to the nuances of the underlying task. Through both quantitative and qualitative experiments across multiple benchmarks, we verify that despite its simplicity, the proposed approach can create detailed and suitable prompts, further boosting the performance of current LLMs.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that when large - language models (LLMs) respond to user queries, the quality of their output is highly dependent on the design of prompt words. Although there are already many strategies to improve prompt words, such as manual design and in - domain optimization, the effectiveness of these methods in unrestricted scenarios is still limited. The former depends on human - designed solutions for specific problems, and the latter usually has poor generalization ability for unseen scenarios. Therefore, this paper proposes a new method - Hierarchical Multi - Agent Workflow (HMAW), aiming to enable LLMs to freely design the best prompt words according to their own situations, thereby improving their performance in various tasks.

Specifically, HMAW contains a hierarchical LLM system. First, prompt words with precise instructions and accurate wording are constructed in a hierarchical manner, and then this prompt word is used to generate the final answer to the user query. Compared with previous work, HMAW does not impose any artificial limitations, does not require training, and is completely task - independent, while being able to adapt to the nuances of the underlying tasks. Through quantitative and qualitative experiments on multiple benchmarks, it has been verified that although the method is simple, the proposed HMAW can create detailed and appropriate prompt words, further enhancing the performance of current LLMs.

In short, this paper mainly solves the problem of poor generalization ability of existing prompt word optimization methods in different tasks, and provides a zero - sample, task - independent, and query - targeted prompt word optimization scheme by introducing HMAW.