SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning

Yuze Zhao,Jintao Huang,Jinghan Hu,Xingjun Wang,Yunlin Mao,Daoze Zhang,Zeyinzi Jiang,Zhikai Wu,Baole Ai,Ang Wang,Wenmeng Zhou,Yingda Chen
2024-08-19
Abstract:Recent development in Large Language Models (LLMs) and Multi-modal Large Language Models (MLLMs) have leverage Attention-based Transformer architectures and achieved superior performance and generalization capabilities. They have since covered extensive areas of traditional learning tasks. For instance, text-based tasks such as text-classification and sequence-labeling, as well as multi-modal tasks like Visual Question Answering (VQA) and Optical Character Recognition (OCR), which were previously addressed using different models, can now be tackled based on one foundation model. Consequently, the training and lightweight fine-tuning of LLMs and MLLMs, especially those based on Transformer architecture, has become particularly important. In recognition of these overwhelming needs, we develop SWIFT, a customizable one-stop infrastructure for large models. With support of over $300+$ LLMs and $50+$ MLLMs, SWIFT stands as the open-source framework that provide the most comprehensive support for fine-tuning large models. In particular, it is the first training framework that provides systematic support for MLLMs. In addition to the core functionalities of fine-tuning, SWIFT also integrates post-training processes such as inference, evaluation, and model quantization, to facilitate fast adoptions of large models in various application scenarios. With a systematic integration of various training techniques, SWIFT offers helpful utilities such as benchmark comparisons among different training techniques for large models. For fine-tuning models specialized in agent framework, we show that notable improvements on the ToolBench leader-board can be achieved by training with customized dataset on SWIFT, with an increase of 5.2%-21.8% in the Act.EM metric over various baseline models, a reduction in hallucination by 1.6%-14.1%, and an average performance improvement of 8%-17%.
Computation and Language
What problem does this paper attempt to address?
The paper aims to address the issues encountered during the training and lightweight fine-tuning of large language models (LLMs) and multimodal large language models (MLLMs), particularly those based on the Transformer architecture. Specifically, the authors have developed SWIFT, a customizable one-stop infrastructure framework designed to support the fine-tuning of large-scale models. SWIFT has the following key features: 1. **Comprehensive support for various models**: Supports over 300 LLMs and more than 50 MLLMs, covering almost all mainstream open-source models, and supports over 150 pure text and multimodal datasets. 2. **Integration of lightweight training techniques**: In addition to the standard Attention structure, it also supports the training and inference of other structures such as the Mamba model, and supports large-scale parallel pre-training of Megatron structure models. 3. **Advanced tuning techniques**: Implements various state-of-the-art (SOTA) tuners, such as LoRA, QLoRA, LongLoRA, etc., to enhance lightweight training effects. 4. **Post-training operation support**: Integrates quantization (e.g., BNB/GPTQ/AWQ), LoRA merging, evaluation (supports over 100 pure text and multimodal evaluation sets), as well as inference and deployment capabilities. 5. **Unified framework design**: Eliminates the gap between pure text LLMs and multimodal LLMs through unified data processing, model templates, and training processes. In summary, SWIFT aims to simplify the training process of LLMs and MLLMs by providing a unified technology chain, reducing the cost of understanding and using these models, and offering the first comprehensive multi-task training and end-to-end solution, especially in the training of multimodal models.