Abstract:Scene text in natural images carry additional semantic information to aid in image classification. Existing methods extract scene text based on simple rules or dictionaries. These methods lack full consideration of the deep understanding of the text and the visual text relationship, and are difficult to judge the semantic accuracy and the relevance of the visual text, thus they perform poorly on image classification tasks. Aiming at the above problems, this paper proposes image classification based on cross modal knowledge learning of scene text (CKLST) method. Scene text in natural images carries additional semantic information to aid in image classification. Existing methods lack full consideration of the deep understanding of the text and the visual text relationship, which results in the difficult to judge the semantic accuracy and the relevance of the visual text. This paper proposes image classification based on Cross modal Knowledge Learning of Scene Text (CKLST) method. CKLST consists of three stages: cross‐modal scene text recognition, text semantic enhancement, and visual‐text feature alignment. In the first stage, multi‐attention is used to extract features layer by layer, and a self‐mask‐based iterative correction strategy is utilized to improve the scene text recognition accuracy. In the second stage, knowledge features are extracted using external knowledge and are fused with text features to enhance text semantic information. In the third stage, CKLST realizes visual‐text feature alignment across attention mechanisms with a similarity matrix, thus the correlation between images and text can be captured to improve the accuracy of the image classification tasks. On Con‐Text dataset, Crowd Activity dataset, Drink Bottle dataset, and Synth Text dataset, CKLST can perform significantly better than other baselines on fine‐grained image classification, with improvements of 3.54%, 5.37%, 3.28%, and 2.81% over the best baseline in mAP, respectively.

Knowledge-Based Scene Text Recognition for Industrial Applications

Background-Insensitive Scene Text Recognition with Text Semantic Segmentation

Scene Text Recognition with Semantics

Revisiting Scene Text Recognition: A Data Perspective.

Knowledge Mining with Scene Text for Fine-Grained Recognition

Multimodal Visual-Semantic Representations Learning for Scene Text Recognition

Deep Neural Network for Semantic-based Text Recognition in Images

Cross‐modal knowledge learning with scene text for fine‐grained image classification

Efficient Backbone Search for Scene Text Recognition

Pushing the Performance Limit of Scene Text Recognizer without Human Annotation

Research on the Text Detection and Recognition in Natural Scene Images

Flexible scene text recognition based on dual attention mechanism

Multi-Granularity Prediction for Scene Text Recognition

An extended attention mechanism for scene text recognition

Towards Accurate Scene Text Recognition with Semantic Reasoning Networks

Enhancing Scene Text Recognition by Strengthening Attention Alignment

A Feasible Framework for Arbitrary-Shaped Scene Text Recognition

Knowledge enhanced graph inference network based entity-relation extraction and knowledge graph construction for industrial domain

Multi-Granularity Prediction with Learnable Fusion for Scene Text Recognition

Instruction-Guided Scene Text Recognition

Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer