Abstract:Scene text in natural images carry additional semantic information to aid in image classification. Existing methods extract scene text based on simple rules or dictionaries. These methods lack full consideration of the deep understanding of the text and the visual text relationship, and are difficult to judge the semantic accuracy and the relevance of the visual text, thus they perform poorly on image classification tasks. Aiming at the above problems, this paper proposes image classification based on cross modal knowledge learning of scene text (CKLST) method. Scene text in natural images carries additional semantic information to aid in image classification. Existing methods lack full consideration of the deep understanding of the text and the visual text relationship, which results in the difficult to judge the semantic accuracy and the relevance of the visual text. This paper proposes image classification based on Cross modal Knowledge Learning of Scene Text (CKLST) method. CKLST consists of three stages: cross‐modal scene text recognition, text semantic enhancement, and visual‐text feature alignment. In the first stage, multi‐attention is used to extract features layer by layer, and a self‐mask‐based iterative correction strategy is utilized to improve the scene text recognition accuracy. In the second stage, knowledge features are extracted using external knowledge and are fused with text features to enhance text semantic information. In the third stage, CKLST realizes visual‐text feature alignment across attention mechanisms with a similarity matrix, thus the correlation between images and text can be captured to improve the accuracy of the image classification tasks. On Con‐Text dataset, Crowd Activity dataset, Drink Bottle dataset, and Synth Text dataset, CKLST can perform significantly better than other baselines on fine‐grained image classification, with improvements of 3.54%, 5.37%, 3.28%, and 2.81% over the best baseline in mAP, respectively.

Multi-scene ancient Chinese text recognition with deep coupled alignments

Cross-Lingual Text Image Recognition Via Multi-Task Sequence to Sequence Learning.

Multiscale Attention-Based Subdomain Dynamic Adaptation for Cross-Domain Scene Classification

An effective multi-modal adaptive contextual feature information fusion method for Chinese long text classification

Joint learning of video scene detection and annotation via multi-modal adaptive context network

Contrastive Learning for Multi-Modal Automatic Code Review

Class-Aware Mask-guided feature refinement for scene text recognition

MCAFNet: Multi-Channel Attention Fusion Network-Based CNN For Remote Sensing Scene Classification

Cross-Scene Joint Classification of Multisource Data With Multilevel Domain Adaption Network

Multi-Grained Cross-modal Alignment for Learning Open-vocabulary Semantic Segmentation from Text Supervision

Masking-Based Cross-Modal Remote Sensing Image–Text Retrieval via Dynamic Contrastive Learning

Chinese Text Recognition with A Pre-Trained CLIP-Like Model Through Image-IDS Aligning

CDistNet: Perceiving Multi-Domain Character Distance for Robust Text Recognition.

Scene Chinese Recognition with Local and Global Attention

Cross‐modal knowledge learning with scene text for fine‐grained image classification

MA-CharNet: Multi-angle fusion character recognition network

Transcending Fusion: A Multi-Scale Alignment Method for Remote Sensing Image-Text Retrieval

Scene-text aware cross-modal retrieval based on semantic matching (ChinaMM2024)

Multi-task Learning for Macromolecule Classification, Segmentation and Coarse Structural Recovery in Cryo-Tomography

MTSTR: Multi-task learning for low-resolution scene text recognition via dual attention mechanism and its application in logistics industry