Abstract:Context-aware methods have achieved remarkable advancements in supervised scene text recognition by leveraging semantic priors from words. Considering the heterogeneity of text and background in STR, we propose that such contextual priors can be reinterpreted as the relations between textual elements, serving as effective self-supervised labels for representation learning. However, textual relations are restricted to the finite size of the dataset due to lexical dependencies, which causes over-fitting problem, thus compromising the representation quality. To address this, our work introduces a unified framework of Relational Contrastive Learning and Masked Image Modeling for STR (RCMSTR), which explicitly models the enriched textual relations. For the RCL branch, we first introduce the relational rearrangement module to cultivate new relations on the fly. Based on this, we further conduct relational contrastive learning to model the intra- and inter-hierarchical relations for frames, sub-words and words. On the other hand, MIM can naturally boost the context information via masking, where we find that the block masking strategy is more effective for STR. For the effective integration of RCL and MIM, we also introduce a novel decoupling design aimed at mitigating the impact of masked images on contrastive learning. Additionally, to enhance the compatibility of MIM with CNNs, we propose the adoption of sparse convolutions and directly sharing the weights with dense convolutions in training. The proposed RCMSTR demonstrates superior performance in various evaluation protocols for different STR-related downstream tasks, outperforming the existing state-of-the-art self-supervised STR techniques. Ablation studies and qualitative experimental results further validate the effectiveness of our method. The code and pre-trained models will be available at <a class="link-external link-https" href="https://github.com/ThunderVVV/RCMSTR" rel="external noopener nofollow">this https URL</a> .

Self-Supervised Learning for Visual Relationship Detection through Masked Bounding Box Reconstruction

Visual relationship detection with a deep convolutional relationship network

Masked Spatio-Temporal Structure Prediction for Self-supervised Learning on Point Cloud Videos

Visual Relationship Detection With Visual-Linguistic Knowledge From Multimodal Representations

Visual Relationship Detection: A Survey

2.5D Visual Relationship Detection

Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection

Towards Latent Masked Image Modeling for Self-Supervised Visual Representation Learning

Masked Modeling for Self-supervised Representation Learning on Vision and Beyond

Detecting Visual Relationships Using Box Attention

Natural Language Guided Visual Relationship Detection

Masked Relation Learning for DeepFake Detection

Adaptively Clustering-Driven Learning for Visual Relationship Detection

Leveraging Auxiliary Text for Deep Recognition of Unseen Visual Relationships

VSRD: Instance-Aware Volumetric Silhouette Rendering for Weakly Supervised 3D Object Detection

Relational Contrastive Learning and Masked Image Modeling for Scene Text Recognition

Visual relationship detection with recurrent attention and negative sampling

Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation

PointUR-RL: Unified Self-Supervised Learning Method Based on Variable Masked Autoencoder for Point Cloud Reconstruction and Representation Learning

A Masked Image Reconstruction Network for Document-level Relation Extraction

Exploring Relations in Untrimmed Videos for Self-Supervised Learning