Concatenated Masked Autoencoders as Spatial-Temporal Learner

Zhouqiang Jiang,Bowen Wang,Tong Xiang,Zhaofeng Niu,Hong Tang,Guangshun Li,Liangzhi Li

2023-12-14

Abstract:Learning representations from videos requires understanding continuous motion and visual correspondences between frames. In this paper, we introduce the Concatenated Masked Autoencoders (CatMAE) as a spatial-temporal learner for self-supervised video representation learning. For the input sequence of video frames, CatMAE keeps the initial frame unchanged while applying substantial masking (95%) to subsequent frames. The encoder in CatMAE is responsible for encoding visible patches for each frame individually; subsequently, for each masked frame, the decoder leverages visible patches from both previous and current frames to reconstruct the original image. Our proposed method enables the model to estimate the motion information between visible patches, match the correspondences between preceding and succeeding frames, and ultimately learn the evolution of scenes. Furthermore, we propose a new data augmentation strategy, Video-Reverse (ViRe), which uses reversed video frames as the model's reconstruction targets. This further encourages the model to utilize continuous motion details and correspondences to complete the reconstruction, thereby enhancing the model's capabilities. Compared to the most advanced pre-training methods, CatMAE achieves a leading level in video segmentation tasks and action recognition tasks.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively capture continuous motion information and inter - frame correspondences in video representation learning. Existing methods have limitations when dealing with the time dimension of videos. They either ignore the differences between spatial and temporal dimensions or are unable to model continuous motion information, resulting in learned representations that are difficult to integrate continuous motion information and inter - frame correspondences simultaneously. For this reason, the paper proposes Concatenated Masked Autoencoders (CatMAE), which enhances the learning ability of the encoder through a concatenated information channel masking strategy, thereby better capturing the long - term dynamic evolution and inter - frame correspondences in videos. In addition, the paper also proposes a new data augmentation strategy - Video - Reverse (ViRe), which uses reversed video frames as the reconstruction target of the model to further encourage the model to utilize continuous motion details and correspondences to complete the reconstruction, thereby enhancing the model's ability. Through these methods, CatMAE has achieved leading - level performance in video segmentation and action recognition tasks.