Abstract:Convolutional neural networks (CNNs) and Transformer architectures have traditionally been recognized as the preferred models for addressing computer vision tasks. However, there has been a recent surge in the popularity of networks based on multi-layer perceptron (MLP) structures that do not rely on convolution or attention mechanisms. These MLP architectures have demonstrated exceptional performance in image classification tasks, exhibiting lower time complexity while maintaining high accuracy. In contrast, video classification tasks involve larger amounts of data and necessitate more intricate feature extraction, resulting in increased time and resource consumption. To enhance computational efficiency and minimize resource utilization, we propose a convolution-free and Transformer-free architecture for video classification called Video-MLP for video classification. Video-MLP utilizes a simple MLP structure to learn video features. Specifically, it comprises two types of layers: Spatial-Mixer and Temporal-Mixer, which respectively capture spatial and temporal information. The Spatial-Mixer extracts spatial information from each frame along the height and width dimensions, while the Temporal-Mixer models temporal information for the same spatial positions across frames. To improve the efficiency of spatial-temporal modeling in our network, we used a spatial-temporal information fusion approach to integrate information at different scales. Additionally, we grouped the input data along the time dimension and designed three different grouping schemes when extracting temporal information. The experimental results indicate that Video-MLP achieved accuracy rates of 87.2% on the Kinetics-400 dataset and 75.3% on the SomethingV2 dataset, outperforming models with equivalent computational complexity. Notably, Video-MLP achieved these results without using convolution and attention mechanisms, and without pre-training on large-scale image and video datasets.

Hire-MLP: Vision MLP Via Hierarchical Rearrangement

ConvMLP: Hierarchical Convolutional MLPs for Vision

Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition

S$^2$-MLP: Spatial-Shift MLP Architecture for Vision

RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality

MLP Architectures for Vision-and-Language Modeling: An Empirical Study

Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs

Rethinking Token-Mixing MLP for MLP-based Vision Backbone

MC-MLP:Multiple Coordinate Frames in all-MLP Architecture for Vision

SpiralMLP: A Lightweight Vision MLP Architecture

DynaMixer: A Vision MLP Architecture with Dynamic Mixing.

CS-Mixer: A Cross-Scale Vision MLP Model with Spatial-Channel Mixing

X-MLP: A Patch Embedding-Free MLP Architecture for Vision

Transformer Vs. MLP-Mixer: Exponential Expressive Gap For NLP Problems

Caterpillar: A Pure-MLP Architecture with Shifted-Pillars-Concatenation

Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP

Video-Mlp: Convolution-Free, Attention-Free Architecture for Video Classification

Full-resolution MLPs Empower Medical Dense Prediction

MAIM: a Mixer MLP Architecture for Image Matching

Vision Transformers with Hierarchical Attention

Multi-Scale MLP-Mixer for image classification