Set-Based Prompting: Provably Solving the Language Model Order Dependency Problem

Reid McIlroy-Young,Katrina Brown,Conlan Olson,Linjun Zhang,Cynthia Dwork
2024-06-12
Abstract:The development of generative language models that can create long and coherent textual outputs via autoregression has lead to a proliferation of uses and a corresponding sweep of analyses as researches work to determine the limitations of this new paradigm. Unlike humans, these 'Large Language Models' (LLMs) are highly sensitive to small changes in their inputs, leading to unwanted inconsistency in their behavior. One problematic inconsistency when LLMs are used to answer multiple-choice questions or analyze multiple inputs is order dependency: the output of an LLM can (and often does) change significantly when sub-sequences are swapped, despite both orderings being semantically identical. In this paper we present Set-Based Prompting, a technique that guarantees the output of an LLM will not have order dependence on a specified set of sub-sequences. We show that this method provably eliminates order dependency, and that it can be applied to any transformer-based LLM to enable text generation that is unaffected by re-orderings. Delving into the implications of our method, we show that, despite our inputs being out of distribution, the impact on expected accuracy is small, where the expectation is over the order of uniformly chosen shuffling of the candidate responses, and usually significantly less in practice. Thus, Set-Based Prompting can be used as a 'dropped-in' method on fully trained models. Finally, we discuss how our method's success suggests that other strong guarantees can be obtained on LLM performance via modifying the input representations.
Computation and Language,Artificial Intelligence,Machine Learning
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the order - dependency issue of large - language models (LLMs) when processing inputs. Specifically, when the order of input subsequences changes, even if these changes have no semantic impact, the output of LLMs will change significantly. This order - dependency not only leads to inconsistent model behavior but also raises issues regarding algorithmic fairness and reliability. For example, when comparing candidates or summarizing medical papers, if changes in the input order result in different output results, this will seriously affect the credibility and fairness of decision - making. To solve this problem, the paper proposes a technique named "Set - Based Prompting". This technique ensures that the output of LLMs will not be affected by changes in the order of these subsequences by removing the order information of specified subsequences from the input. The paper proves that this method can effectively eliminate order - dependency and is applicable to any LLMs based on the Transformer architecture. In addition, the study also explores the impact of this method on model performance. The results show that although the input data is out - of - distribution, the impact on the expected accuracy rate is small, and in general, the actual impact is even smaller. Therefore, set - based prompting can be applied as a "plug - and - play" method to fully - trained models to improve their reliability and consistency.