Abstract:We introduce Bonito, an open-source model for conditional task generation that converts unannotated text into task-specific training datasets for instruction tuning. We aim to enable zero-shot task adaptation of large language models on users' specialized, private data. We train Bonito by fine-tuning a pretrained large language model on a new large-scale dataset with 1.65M examples created by remixing existing instruction tuning datasets into meta-templates. The meta-templates for a dataset produce training examples where the input is the unannotated text and the task attribute and the output consists of the instruction and the response. We use Bonito to generate synthetic tasks for seven datasets from specialized domains with unannotated text across three task types -- yes-no question answering, extractive question answering, and natural language inference -- and adapt language models. We show that Bonito significantly improves the average performance of pretrained and instruction tuned models over the de facto self supervised baseline. For example, adapting Mistral-Instruct-v2 and instruction tuned variants of Mistral and Llama2 with Bonito improves the strong zero-shot performance by 22.1 F1 points whereas the next word prediction objective undoes some of the benefits of instruction tuning and reduces the average performance by 0.8 F1 points. We conduct additional experiments with Bonito to understand the effects of the domain, the size of the training set, and the choice of alternative synthetic task generators. Overall, we show that learning with synthetic instruction tuning datasets is an effective way to adapt language models to new domains. The model, dataset, and code are available at <a class="link-external link-https" href="https://github.com/BatsResearch/bonito" rel="external noopener nofollow">this https URL</a>.

Finetuned Language Models Are Zero-Shot Learners

Scaling Instruction-Finetuned Language Models

MLAN: Language-Based Instruction Tuning Improves Zero-Shot Generalization of Multimodal Large Language Models

Evaluating the Zero-shot Robustness of Instruction-tuned Language Models

On Instruction-Finetuning Neural Machine Translation Models

A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Tasks

Deep Exploration of Cross-Lingual Zero-Shot Generalization in Instruction Tuning

Maybe Only 0.5 Training Data Instruction Tuning

InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models

An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models

Enhancing Zero-Shot Crypto Sentiment With Fine-Tuned Language Model and Prompt Engineering

Non-instructional Fine-tuning: Enabling Instruction-Following Capabilities in Pre-trained Language Models without Instruction-Following Data

LIMIT: Less Is More for Instruction Tuning Across Evaluation Paradigms

Phased Instruction Fine-Tuning for Large Language Models

Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation

Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation

M^2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning

MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning

Go-tuning: Improving Zero-shot Learning Abilities of Smaller Language Models

Zero-shot cross-lingual transfer in instruction tuning of large language models

Instruction Tuning with GPT-4