Abstract:Scene text spotting is of great importance to the computer vision community due to its wide variety of applications. Recent methods attempt to introduce linguistic knowledge for challenging recognition rather than pure visual classification. However, how to effectively model the linguistic rules in end-to-end deep networks remains a research challenge. In this paper, we argue that the limited capacity of language models comes from 1) implicit language modeling; 2) unidirectional feature representation; and 3) language model with noise input. Correspondingly, we propose an autonomous, bidirectional and iterative ABINet++ for scene text spotting. First, the autonomous suggests enforcing explicitly language modeling by decoupling the recognizer into vision model and language model and blocking gradient flow between both models. Second, a novel bidirectional cloze network (BCN) as the language model is proposed based on bidirectional feature representation. Third, we propose an execution manner of iterative correction for the language model which can effectively alleviate the impact of noise input. Additionally, based on an ensemble of the iterative predictions, a self-training method is developed which can learn from unlabeled images effectively. Finally, to polish ABINet++ in long text recognition, we propose to aggregate horizontal features by embedding Transformer units inside a U-Net, and design a position and content attention module which integrates character order and content to attend to character features precisely. ABINet++ achieves state-of-the-art performance on both scene text recognition and scene text spotting benchmarks, which consistently demonstrates the superiority of our method in various environments especially on low-quality images. Besides, extensive experiments including in English and Chinese also prove that, a text spotter that incorporates our language modeling method can significantly improve its performance both in accuracy and speed compared with commonly used attention-based recognizers. Code is available at https://github.com/FangShancheng/ABINet-PP.

Adaptive Importance Pooling Network for Scene Text Recognition

Scene Text Detection and Recognition System for Visually Impaired People in Real World

Deep Neural Network with Attention Model for Scene Text Recognition.

Scene Text Recognition with Cascade Attention Network.

Flexible scene text recognition based on dual attention mechanism

A pooling based scene text proposal technique for scene text reading in the wild

HAFE: A Hierarchical Awareness and Feature Enhancement Network for Scene Text Recognition

SVTR-SRNet: A Deep Learning Model for Scene Text Recognition via SVTR Framework and Spatial Reduction Mechanism

MASTER: Multi-Aspect Non-local Network for Scene Text Recognition

ReADS: A Rectified Attentional Double Supervised Network for Scene Text Recognition

Hierarchical Refined Attention for Scene Text Recognition.

Efficient Backbone Search for Scene Text Recognition

IPAD: Iterative, Parallel, and Diffusion-based Network for Scene Text Recognition

A Feasible Framework for Arbitrary-Shaped Scene Text Recognition

A New Perspective for Flexible Feature Gathering in Scene Text Recognition Via Character Anchor Pooling

SVIPTR: Fast and Efficient Scene Text Recognition with Vision Permutable Extractor

STAR-Net: A SpaTial Attention Residue Network for Scene Text Recognition

Character Region Awareness Network for Scene Text Recognition

ABINet++: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Spotting

A holistic representation guided attention network for scene text recognition

Robust Scene Text Recognition Through Adaptive Image Enhancement