Abstract:The Multimodal Large Language Models (MLLMs) have activated the capabilitiesof Large Language Models (LLMs) in solving visual-language tasks by integratingvisual information. The prevailing approach in existing MLLMs involvesemploying an image encoder to extract visual features, converting thesefeatures into visual tokens via an adapter, and then integrating them with theprompt into the LLM. However, because the process of image encoding isprompt-agnostic, the extracted visual features only provide a coarsedescription of the image, impossible to focus on the requirements of theprompt. On one hand, it is easy for image features to lack information aboutthe prompt-specified objects, resulting in unsatisfactory responses. On theother hand, the visual features contain a large amount of irrelevantinformation, which not only increases the burden on memory but also worsens thegeneration effectiveness. To address the aforementioned issues, we propose\textbf{PIP-MM}, a framework that \textbf{P}re-\textbf{I}ntegrates\textbf{P}rompt information into the visual encoding process using existingmodules of MLLMs. Specifically, We utilize the frozen LLM in the MLLM tovectorize the input prompt, which summarizes the requirements of the <a class="link-external link-http" href="http://prompt.Then" rel="external noopener nofollow">this http URL</a>, we input the prompt vector into our trained Multi-Layer Perceptron (MLP)to align with the visual input requirements, and subsequently replace the classembedding in the image encoder. Since our model only requires adding atrainable MLP, it can be applied to any MLLM. To validate the effectiveness ofPIP-MM, we conducted experiments on multiple benchmarks. Automated evaluationmetrics and manual assessments demonstrate the strong performance of <a class="link-external link-http" href="http://PIP-MM.Particularly" rel="external noopener nofollow">this http URL</a> noteworthy is that our model maintains excellent generationresults even when half of the visual tokens are reduced.

Memory-Inspired Temporal Prompt Interaction for Text-Image Classification

ModalPrompt:Dual-Modality Guided Prompt for Continual Learning of Large Multimodal Models

Modality-invariant and Specific Prompting for Multimodal Human Perception Understanding

Adaptive Multi-Modality Prompt Learning

Deep Correlated Prompting for Visual Recognition with Missing Modalities

Mutual Prompt Leaning for Vision Language Models

Modular and Parameter-Efficient Multimodal Fusion with Prompting

Instruction-ViT: Multi-modal prompts for instruction learning in vision transformer

MuDPT: Multi-modal Deep-symphysis Prompt Tuning for Large Pre-trained Vision-Language Models

Understanding the Multi-modal Prompts of the Pre-trained Vision-Language Model

RetrievalMMT: Retrieval-Constrained Multi-Modal Prompt Learning for Multi-Modal Machine Translation

Efficient Multimodal Fusion Via Interactive Prompting

PIP-MM: Pre-Integrating Prompt Information into Visual Encoding via Existing MLLM Structures

Instruction-ViT: Multi-Modal Prompts for Instruction Learning in ViT

Modal-aware Prompt Tuning with Deep Adaptive Feature Enhancement

Multi-Prompt with Depth Partitioned Cross-Modal Learning

MPT4LM: Multi-Modal Prompt Tuning Makes Pre-Trained Large Language Models Better Vision-Language Learners

Progressive Multi-modal Conditional Prompt Tuning

Multimodal Prompting with Missing Modalities for Visual Recognition

Rethinking Visual Prompt Learning as Masked Visual Token Modeling

Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning