Abstract:With the flourishing of social media platforms, vision-language pre-training (VLP) recently has received great attention and many remarkable progresses have been achieved. The success of VLP largely benefits from the information complementation and enhancement between different modalities. However, most of recent studies focus on cross-modal contrastive learning (CMCL) to promote image-text alignment by pulling embeddings of positive sample pairs together while pushing those of negative pairs apart, which ignores the natural asymmetry property between different modalities and requires large-scale image-text corpus to achieve arduous progress. To mitigate this predicament, we propose CMAL, a Cross-Modal Associative Learning framework with anchor points detection and cross-modal associative learning for VLP. Specifically, we first respectively embed visual objects and textual tokens into separate hypersphere spaces to learn intra-modal hidden features, and then design a cross-modal associative prompt layer to perform anchor point masking and swap feature filling for constructing a hybrid cross-modal associative prompt. Afterwards, we exploit a unified semantic encoder to learn their cross-modal interactive features for context adaptation. Finally, we design an associative mapping classification layer to learn potential associative mappings between modalities at anchor points, within which we develop a fresh self-supervised associative mapping classification task to boost CMAL's performance. Experimental results verify the effectiveness of CMAL, showing that it achieves competitive performance against previous CMCL-based methods on four common downstream vision-and-language tasks, with significantly fewer corpus. Especially, CMAL obtains new state-of-the-art results on SNLI-VE and REC (testA).

AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model

eP-ALM: Efficient Perceptual Augmentation of Language Models

Spider: Any-to-Many Multimodal LLM

Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models

CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training

mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration

MMMModal -- Multi-Images Multi-Audio Multi-turn Multi-Modal

TextMI: Textualize Multimodal Information for Integrating Non-verbal Cues in Pre-trained Language Models

PaLM-E: An Embodied Multimodal Language Model

AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling

Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception

Language Is Not All You Need: Aligning Perception with Language Models

Large AI Model Empowered Multimodal Semantic Communications

CaMML: Context-Aware Multimodal Learner for Large Models

Read and Think: An Efficient Step-wise Multimodal Language Model for Document Understanding and Reasoning

EMMA: Efficient Visual Alignment in Multi-Modal LLMs

Retrieval-Augmented Multimodal Language Modeling

Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Models

EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model

MINER: Mining the Underlying Pattern of Modality-Specific Neurons in Multimodal Large Language Models