Abstract:Following multiple instructions is a crucial ability for large language models (LLMs). Evaluating this ability comes with significant challenges: (i) limited coherence between multiple instructions, (ii) positional bias where the order of instructions affects model performance, and (iii) a lack of objectively verifiable tasks. To address these issues, we introduce a benchmark designed to evaluate models' abilities to follow multiple instructions through sequential instruction following (SIFo) tasks. In SIFo, the successful completion of multiple instructions is verifiable by examining only the final instruction. Our benchmark evaluates instruction following using four tasks (text modification, question answering, mathematics, and security rules), each assessing different aspects of sequential instruction following. Our evaluation of popular LLMs, both closed-source and open-source, shows that more recent and larger models significantly outperform their older and smaller counterparts on the SIFo tasks, validating the benchmark's effectiveness. All models struggle with following sequences of instructions, hinting at an important lack of robustness of today's language models.

What problem does this paper attempt to address?

The paper attempts to address the problem of evaluating the capabilities of large language models (LLMs) in performing multi-instruction sequence tasks. Current methods for evaluating LLMs' instruction-following abilities mainly focus on single instructions, overlooking several key challenges in multi-instruction sequence tasks: 1. **Limited coherence between instructions**: Instructions in a multi-instruction sequence often lack intrinsic connections, making it difficult for the model to understand or execute the entire sequence. 2. **Positional bias**: The order of instructions affects the model's performance, meaning the model may perform differently depending on the position of the instructions in the sequence. 3. **Lack of objectively verifiable tasks**: Existing evaluation methods lack tasks that can objectively verify whether the model has correctly executed the multi-instruction sequence. To address these issues, the paper proposes a new benchmark—SIFo (Sequential Instruction Following), aimed at evaluating the model's ability to perform multi-instruction sequence tasks. The SIFo benchmark includes four tasks: text modification, question answering, mathematics, and safety rules. These tasks are designed to be objectively verifiable, and each task emphasizes different aspects such as text understanding, reasoning, and adherence to safety rules. Through these tasks, the paper evaluates several state-of-the-art LLMs, including both closed-source and open-source models, finding that newer and larger models perform significantly better on SIFo tasks than older and smaller models, but all models still struggle with handling long instruction sequences.

The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models

LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios

Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following

FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models

InFoBench: Evaluating Instruction Following Ability in Large Language Models

FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models

Benchmarking Large Language Models on Controllable Generation under Diversified Instructions

Can Large Language Models Understand Real-World Complex Instructions?

Evaluating Large Language Models at Evaluating Instruction Following

Benchmarking Complex Instruction-Following with Multiple Constraints Composition

Beyond Instruction Following: Evaluating Inferential Rule Following of Large Language Models

Large Language Model Instruction Following: A Survey of Progresses and Challenges

CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models

Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?

Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models

Improving Translation Faithfulness of Large Language Models via Augmenting Instructions

AQA-Bench: An Interactive Benchmark for Evaluating LLMs' Sequential Reasoning Ability

Diverse and Fine-Grained Instruction-Following Ability Exploration with Synthetic Data

Evaluating the Instruction-following Abilities of Language Models using Knowledge Tasks

Fine-tuning Large Language Models with Sequential Instructions

Mixture-of-Instructions: Comprehensive Alignment of a Large Language Model through the Mixture of Diverse System Prompting Instructions