Abstract:Text-rich images have significant and extensive value, deeply integrated into various aspects of human life. Notably, both visual cues and linguistic symbols in text-rich images play crucial roles in information transmission but are accompanied by diverse challenges. Therefore, the efficient and effective understanding of text-rich images is a crucial litmus test for the capability of Vision-Language Models. We have crafted an efficient vision-language model, StrucTexTv3, tailored to tackle various intelligent tasks for text-rich images. The significant design of StrucTexTv3 is presented in the following aspects: Firstly, we adopt a combination of an effective multi-scale reduced visual transformer and a multi-granularity token sampler (MG-Sampler) as a visual token generator, successfully solving the challenges of high-resolution input and complex representation learning for text-rich images. Secondly, we enhance the perception and comprehension abilities of StrucTexTv3 through instruction learning, seamlessly integrating various text-oriented tasks into a unified framework. Thirdly, we have curated a comprehensive collection of high-quality text-rich images, abbreviated as TIM-30M, encompassing diverse scenarios like incidental scenes, office documents, web pages, and screenshots, thereby improving the robustness of our model. Our method achieved SOTA results in text-rich image perception tasks, and significantly improved performance in comprehension tasks. Among multimodal models with LLM decoder of approximately 1.8B parameters, it stands out as a leader, which also makes the deployment of edge devices feasible. In summary, the StrucTexTv3 model, featuring efficient structural design, outstanding performance, and broad adaptability, offers robust support for diverse intelligent application tasks involving text-rich images, thus exhibiting immense potential for widespread application.

StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-training

MaskOCR: Text Recognition with Masked Encoder-Decoder Pretraining

Maskstr: Guide Scene Text Recognition Models with Masking

StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyond

Masked Spatio-Temporal Structure Prediction for Self-supervised Learning on Point Cloud Videos

Masked Text Modeling: A Self-Supervised Pre-training Method for Scene Text Detection

SimVTP: Simple Video Text Pre-training with Masked Autoencoders

LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding

PreSTU: Pre-Training for Scene-Text Understanding

Leveraging Text Localization for Scene Text Removal via Text-aware Masked Image Modeling

Masked Contrastive Pre-Training for Efficient Video-Text Retrieval

Sequence-to-Sequence Pre-training with Unified Modality Masking for Visual Document Understanding

Stacked Convolutional Deep Encoding Network for Video-Text Retrieval.

MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining

Self-Supervised Pre-training with Symmetric Superimposition Modeling for Scene Text Recognition

Train No Evil: Selective Masking for Task-Guided Pre-Training

MaskViT: Masked Visual Pre-Training for Video Prediction

ViTEraser: Harnessing the Power of Vision Transformers for Scene Text Removal with SegMIM Pretraining

Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning

LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking

MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval