Abstract:Large language models (LLMs) have demonstrated impressive zero-shot abilities on a variety of open-ended tasks, while recent research has also explored the use of LLMs for multi-modal generation. In this study, we introduce mPLUG-Owl, a novel training paradigm that equips LLMs with multi-modal abilities through modularized learning of foundation LLM, a visual knowledge module, and a visual abstractor module. This approach can support multiple modalities and facilitate diverse unimodal and multimodal abilities through modality collaboration. The training paradigm of mPLUG-Owl involves a two-stage method for aligning image and text, which learns visual knowledge with the assistance of LLM while maintaining and even improving the generation abilities of LLM. In the first stage, the visual knowledge module and abstractor module are trained with a frozen LLM module to align the image and text. In the second stage, language-only and multi-modal supervised datasets are used to jointly fine-tune a low-rank adaption (LoRA) module on LLM and the abstractor module by freezing the visual knowledge module. We carefully build a visually-related instruction evaluation set OwlEval. Experimental results show that our model outperforms existing multi-modal models, demonstrating mPLUG-Owl's impressive instruction and visual understanding ability, multi-turn conversation ability, and knowledge reasoning ability. Besides, we observe some unexpected and exciting abilities such as multi-image correlation and scene text understanding, which makes it possible to leverage it for harder real scenarios, such as vision-only document comprehension. Our code, pre-trained model, instruction-tuned models, and evaluation set are available at <a class="link-external link-https" href="https://github.com/X-PLUG/mPLUG-Owl" rel="external noopener nofollow">this https URL</a>. The online demo is available at <a class="link-external link-https" href="https://www.modelscope.cn/studios/damo/mPLUG-Owl" rel="external noopener nofollow">this https URL</a>.

Mplug-Octopus: the Versatile Assistant Empowered by A Modularized End-to-End Multimodal LLM

mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration

mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality

mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video

Can Large Language Models Be Good Companions? An LLM-Based Eyewear System with Conversational Common Ground

Multi-User Chat Assistant (MUCA): a Framework Using LLMs to Facilitate Group Conversations

PILL: Plug Into LLM with Adapter Expert and Attention Gate

ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Human

ChatSpot: Bootstrapping Multimodal LLMs Via Precise Referring Instruction Tuning

Prompted LLMs as Chatbot Modules for Long Open-domain Conversation

mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models

Modality Plug-and-Play: Elastic Modality Adaptation in Multimodal LLMs for Embodied AI

Chat with the Environment: Interactive Multimodal Perception Using Large Language Models

User Interaction Patterns and Breakdowns in Conversing with LLM-Powered Voice Assistants

Modular Conversational Agents for Surveys and Interviews

Octopus: Embodied Vision-Language Programmer from Environmental Feedback

Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents

When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration

Towards an LLM-Based Speech Interface for Robot-Assisted Feeding

M^2Chat: Empowering VLM for Multimodal LLM Interleaved Text-Image Generation

Enabling Conversational Interaction with Mobile UI using Large Language Models