Abstract:Context-aware methods have achieved remarkable advancements in supervised scene text recognition by leveraging semantic priors from words. Considering the heterogeneity of text and background in STR, we propose that such contextual priors can be reinterpreted as the relations between textual elements, serving as effective self-supervised labels for representation learning. However, textual relations are restricted to the finite size of the dataset due to lexical dependencies, which causes over-fitting problem, thus compromising the representation quality. To address this, our work introduces a unified framework of Relational Contrastive Learning and Masked Image Modeling for STR (RCMSTR), which explicitly models the enriched textual relations. For the RCL branch, we first introduce the relational rearrangement module to cultivate new relations on the fly. Based on this, we further conduct relational contrastive learning to model the intra- and inter-hierarchical relations for frames, sub-words and words. On the other hand, MIM can naturally boost the context information via masking, where we find that the block masking strategy is more effective for STR. For the effective integration of RCL and MIM, we also introduce a novel decoupling design aimed at mitigating the impact of masked images on contrastive learning. Additionally, to enhance the compatibility of MIM with CNNs, we propose the adoption of sparse convolutions and directly sharing the weights with dense convolutions in training. The proposed RCMSTR demonstrates superior performance in various evaluation protocols for different STR-related downstream tasks, outperforming the existing state-of-the-art self-supervised STR techniques. Ablation studies and qualitative experimental results further validate the effectiveness of our method. The code and pre-trained models will be available at <a class="link-external link-https" href="https://github.com/ThunderVVV/RCMSTR" rel="external noopener nofollow">this https URL</a> .
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: How to make full use of the characteristics of text images in Scene Text Recognition (STR) by using self - supervised learning methods, so as to learn effective feature representations without manual annotation.
Specifically, most of the existing contrastive learning and masked image modeling methods are designed for natural images and have poor performance when directly applied to scene text images. This is because scene text images have the following unique characteristics:
1. **Heterogeneity of text foreground and background**: Text recognition mainly depends on the text rather than the background.
2. **Most text images have only one main direction**: For example, English is read from left to right.
3. **Text images contain character sequences and phrase structures**: These structures have hierarchical relationships at different granularities.
To solve these problems, the author proposes a unified framework named RCMSTR (Relational Contrastive Learning and Masked Image Modeling for Scene Text Recognition), which combines Relational Contrastive Learning (RCL) and Masked Image Modeling (MIM). Its core objective is to improve the self - supervised representation learning of STR in the following ways:
- **Relational Contrastive Learning (RCL)**: Generate new contextual relationships by rearranging images and explicitly model the hierarchical relationships between frames, sub - words and words to enrich text relationships.
- **Masked Image Modeling (MIM)**: Enhance the learning of local and global relationships through block - masking strategies, and design a decoupling framework to improve compatibility with CNN and ViT architectures.
Through these methods, RCMSTR aims to overcome the limitations of existing SSL methods on scene text images, provide more effective feature representations, and thus improve the performance of downstream tasks.
### Formula Summary
1. **InfoNCE Loss Function**:
\[
L_{\text{info}}(q, p, n)=-\log\frac{\exp(q\cdot p / \tau_{\text{info}})}{\sum_{u\in\{n_k\}_{k = 1}^K\cup\{p\}}\exp(q\cdot u / \tau_{\text{info}})}
\]
2. **KL Divergence Loss Function**:
\[
Q_i(q, n)=\frac{\exp(q\cdot n_i / \tau_{\text{kl}})}{\sum_{k = 1}^K\exp(q\cdot n_k / \tau_{\text{kl}})}
\]
\[
P_i(p, n)=\frac{\exp(p\cdot n_i / \tau_{\text{kl}})}{\sum_{k = 1}^K\exp(p\cdot n_k / \tau_{\text{kl}})}
\]
\[
L_{\text{kl}}(q, p, n)=\frac{1}{2}D_{\text{KL}}(P\|Q)+\frac{1}{2}D_{\text{KL}}(Q\|P)
\]
3. **Total Relational Loss**:
\[
L_{\text{re}}(q, p, n)=L_{\text{info}}(q, p, n)+\alpha L_{\text{kl}}(q, p, n)
\]
4. **Relational Contrastive Loss with Augmented Images**:
\[
L_{\text{ere}}(q, p, n)=\frac{1}{2}L_{\text{re}}(q, p, n)+\frac{1}{2}L_{\text{re}}(q_e, p, n)
\]
5. **Hierarchical Relational Contrastive Loss**:
\[