Abstract:Despite the effectiveness of vision-language supervised fine-tuning in enhancing the performance of Vision Large Language Models (VLLMs). However, existing visual instruction tuning datasets include the following limitations: (1) Instruction annotation quality: despite existing VLLMs exhibiting strong performance, instructions generated by those advanced VLLMs may still suffer from inaccuracies, such as hallucinations. (2) Instructions and image diversity: the limited range of instruction types and the lack of diversity in image data may impact the model's ability to generate diversified and closer to real-world scenarios outputs. To address these challenges, we construct a high-quality, diverse visual instruction tuning dataset MMInstruct, which consists of 973K instructions from 24 domains. There are four instruction types: Judgement, Multiple-Choice, Long Visual Question Answering and Short Visual Question Answering. To construct MMInstruct, we propose an instruction generation data engine that leverages GPT-4V, GPT-3.5, and manual correction. Our instruction generation engine enables semi-automatic, low-cost, and multi-domain instruction generation at 1/6 the cost of manual construction. Through extensive experiment validation and ablation experiments, we demonstrate that MMInstruct could significantly improve the performance of VLLMs, e.g., the model fine-tuning on MMInstruct achieves new state-of-the-art performance on 10 out of 12 benchmarks. The code and data shall be available at <a class="link-external link-https" href="https://github.com/yuecao0119/MMInstruct" rel="external noopener nofollow">this https URL</a>.

Offline Prompt Polishing for Low Quality Instructions

Exploring Format Consistency for Instruction Tuning

InstructCoder: Instruction Tuning Large Language Models for Code Editing

MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity

InstructionGPT-4: A 200-Instruction Paradigm for Fine-Tuning MiniGPT-4

Non-instructional Fine-tuning: Enabling Instruction-Following Capabilities in Pre-trained Language Models without Instruction-Following Data

Instruction Tuning for Large Language Models: A Survey

Demystifying Instruction Mixing for Fine-tuning Large Language Models

LIPT: Improving Prompt Tuning with Late Inception Reparameterization

InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction.

X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions

FIPO: Free-form Instruction-oriented Prompt Optimization with Preference Dataset and Modular Fine-tuning Schema

IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization

Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor

PILLOW: Enhancing Efficient Instruction Fine-tuning via Prompt Matching

INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning

Towards Robust Instruction Tuning on Multimodal Large Language Models

BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing

EasyInstruct: An Easy-to-use Instruction Processing Framework for Large Language Models

LIMIT: Less Is More for Instruction Tuning Across Evaluation Paradigms

Raw Text is All you Need: Knowledge-intensive Multi-turn Instruction Tuning for Large Language Model