Abstract:We initiate the first empirical study on the use of MLP architectures for vision-and-language (VL) fusion. Through extensive experiments on 5 VL tasks and 5 robust VQA benchmarks, we find that: (i) Without pre-training, using MLPs for multimodal fusion has a noticeable performance gap compared to transformers; (ii) However, VL pre-training can help close the performance gap; (iii) Instead of heavy multi-head attention, adding tiny one-head attention to MLPs is sufficient to achieve comparable performance to transformers. Moreover, we also find that the performance gap between MLPs and transformers is not widened when being evaluated on the harder robust VQA benchmarks, suggesting using MLPs for VL fusion can generalize roughly to a similar degree as using transformers. These results hint that MLPs can effectively learn to align vision and text features extracted from lower-level encoders without heavy reliance on self-attention. Based on this, we ask an even bolder question: can we have an all-MLP architecture for VL modeling, where both VL fusion and the vision encoder are replaced with MLPs? Our result shows that an all-MLP VL model is sub-optimal compared to state-of-the-art full-featured VL models when both of them get pre-trained. However, pre-training an all-MLP can surprisingly achieve a better average score than full-featured transformer models without pre-training. This indicates the potential of large-scale pre-training of MLP-like architectures for VL modeling and inspires the future research direction on simplifying well-established VL modeling with less inductive design bias. Our code is publicly available at: <a class="link-external link-https" href="https://github.com/easonnie/mlp-vil" rel="external noopener nofollow">this https URL</a>

SepMLP: An All-MLP Architecture for Music Source Separation

Music Source Separation With Band-Split RNN

Music Source Separation Based on a Lightweight Deep Learning Framework (DTTNET: DUAL-PATH TFC-TDF UNET)

Decoupling Magnitude and Phase Estimation with Deep ResUNet for Music Source Separation

Music Source Separation with Band-Split RoPE Transformer

Class-conditional Embeddings for Music Source Separation

Music source separation conditioned on 3D point clouds

Audio query-based music source separation

The Whole Is Greater than the Sum of Its Parts: Improving Music Source Separation by Bridging Network

The whole is greater than the sum of its parts: improving music source separation by bridging networks

MLP Architectures for Vision-and-Language Modeling: An Empirical Study

A Two-Stage Band-Split Mamba-2 Network For Music Separation

A Unified Model for Zero-shot Music Source Separation, Transcription and Synthesis

Hierarchic Temporal Convolutional Network With Cross-Domain Encoder for Music Source Separation

Multi-scale temporal-frequency attention for music source separation

SCNet: Sparse Compression Network for Music Source Separation

MC-MLP:Multiple Coordinate Frames in all-MLP Architecture for Vision

D3Net: Densely connected multidilated DenseNet for music source separation

Towards Musically Meaningful Explanations Using Source Separation

Fusionmlp: A Mlp-Based Unified Image Fusion Framework