Abstract:Contrastive learning (CL) for Vision Transformers (ViTs) in image domains has achieved performance comparable to CL for traditional convolutional backbones. However, in 3D point cloud pretraining with ViTs, masked autoencoder (MAE) modeling remains dominant. This raises the question: Can we take the best of both worlds? To answer this question, we first empirically validate that integrating MAE-based point cloud pre-training with the standard contrastive learning paradigm, even with meticulous design, can lead to a decrease in performance. To address this limitation, we reintroduce CL into the MAE-based point cloud pre-training paradigm by leveraging the inherent contrastive properties of MAE. Specifically, rather than relying on extensive data augmentation as commonly used in the image domain, we randomly mask the input tokens twice to generate contrastive input pairs. Subsequently, a weight-sharing encoder and two identically structured decoders are utilized to perform masked token reconstruction. Additionally, we propose that for an input token masked by both masks simultaneously, the reconstructed features should be as similar as possible. This naturally establishes an explicit contrastive constraint within the generative MAE-based pre-training paradigm, resulting in our proposed method, Point-CMAE. Consequently, Point-CMAE effectively enhances the representation quality and transfer performance compared to its MAE counterpart. Experimental evaluations across various downstream applications, including classification, part segmentation, and few-shot learning, demonstrate the efficacy of our framework in surpassing state-of-the-art techniques under standard ViTs and single-modal settings. The source code and trained models are available at: <a class="link-external link-https" href="https://github.com/Amazingren/Point-CMAE" rel="external noopener nofollow">this https URL</a>.

TS-MVP: Time-Series Representation Learning by Multi-view Prototypical Contrastive Learning.

Multi-view Self-Supervised Contrastive Learning for Multivariate Time Series

Towards Unsupervised Time Series Representation Learning: A Decomposition Perspective

Ti-MAE: Self-Supervised Masked Time Series Autoencoders

Multi-Task Self-Supervised Time-Series Representation Learning

Time-Series Representation Learning via Temporal and Contextual Contrasting

Self-Supervised Contrastive Representation Learning for Semi-Supervised Time-Series Classification

A Deep Multi-Task Representation Learning Method for Time Series Classification and Retrieval.

Time Series Representation Learning with Supervised Contrastive Temporal Transformer

Time-series representation learning via Time-Frequency Fusion Contrasting

TS-TWC: A time series representation learning framework based on Time-Wavelet contrasting

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

Distillation Enhanced Time Series Forecasting Network with Momentum Contrastive Learning

MHCCL: Masked Hierarchical Cluster-Wise Contrastive Learning for Multivariate Time Series

Parametric Augmentation for Time Series Contrastive Learning

Time Series Classification Based on Data-Augmented Contrastive Learning.

Simple Contrastive Representation Learning for Time Series Forecasting

Multi-Scale and Multi-Modal Contrastive Learning Network for Biomedical Time Series

Bringing Masked Autoencoders Explicit Contrastive Properties for Point Cloud Self-Supervised Learning