Advances of Scene Text Datasets

Masakazu Iwamura

DOI: https://doi.org/10.48550/arXiv.1812.05219

2018-12-13

Computer Vision and Pattern Recognition

Abstract:This article introduces publicly available datasets in scene text detection and recognition. The information is as of 2017.

What problem does this paper attempt to address?

Scene text detection and recognition with advances in deep learning: a survey

Xiyan Liu,Gaofeng Meng,Chunhong Pan

DOI: https://doi.org/10.1007/s10032-019-00320-5

2019-03-27

International Journal on Document Analysis and Recognition (IJDAR)

Abstract:Scene text detection and recognition has become a very active research topic in recent several years. It can find many applications in reality ranging from navigation for vision-impaired people to semantic natural scene understanding. In this survey, we are intended to give a thorough and in-depth reviews on the recent advances on this topic, mainly focusing on the methods that appeared in the past 5 years for text detection and recognition in images and videos, including the recent state-of-the-art techniques on the following three related topics: (1) scene text detection, (2) scene text recognition and (3) end-to-end text recognition system. Compared with the previous survey, this survey pays more attention to the application of deep learning techniques on scene text detection and recognition. We also give a brief introduction of other related works such as script identification, text/non-text classification and text-to-image retrieval. This survey also reviews and summarizes some benchmark datasets that are widely used in the literature. Based on these datasets, performances of state-of-the-art approaches are shown and discussed. Finally, we conclude this survey by pointing out several potential directions on scene text detection and recognition that need to be well explored in the future.
Scene text detection and recognition: recent advances and future trends

Yingying Zhu,Cong Yao,Xiang Bai

DOI: https://doi.org/10.1007/s11704-015-4488-0

IF: 2.6688

2015-06-22

Frontiers of Computer Science

Abstract:Text, as one of the most influential inventions of humanity, has played an important role in human life, so far from ancient times. The rich and precise information embodied in text is very useful in a wide range of vision-based applications, therefore text detection and recognition in natural scenes have become important and active research topics in computer vision and document analysis. Especially in recent years, the community has seen a surge of research efforts and substantial progresses in these fields, though a variety of challenges (e.g. noise, blur, distortion, occlusion and variation) still remain. The purposes of this survey are three-fold: 1) introduce up-to-date works, 2) identify state-of-the-art algorithms, and 3) predict potential research directions in the future. Moreover, this paper provides comprehensive links to publicly available resources, including benchmark datasets, source codes, and online demos. In summary, this literature review can serve as a good reference for researchers in the areas of scene text detection and recognition.

computer science, information systems, theory & methods, software engineering
Scene Text Detection and Recognition: The Deep Learning Era

Shangbang Long,Xin He,Cong Yao

DOI: https://doi.org/10.1007/s11263-020-01369-0

IF: 13.369

2020-08-27

International Journal of Computer Vision

Abstract:With the rise and development of deep learning, computer vision has been tremendously transformed and reshaped. As an important research area in computer vision, scene text detection and recognition has been inevitably influenced by this wave of revolution, consequentially entering the era of deep learning. In recent years, the community has witnessed substantial advancements in mindset, methodology and performance. This survey is aimed at summarizing and analyzing the major changes and significant progresses of scene text detection and recognition in the deep learning era. Through this article, we devote to: (1) introduce new insights and ideas; (2) highlight recent techniques and benchmarks; (3) look ahead into future trends. Specifically, we will emphasize the dramatic differences brought by deep learning and remaining grand challenges. We expect that this review paper would serve as a reference book for researchers in this field. Related resources are also collected in our Github repository (<a href="https://github.com/Jyouhou/SceneTextPapers">https://github.com/Jyouhou/SceneTextPapers</a>).

computer science, artificial intelligence
The First Swahili Language Scene Text Detection and Recognition Dataset

Fadila Wendigoundi Douamba,Jianjun Song,Ling Fu,Yuliang Liu,Xiang Bai

2024-05-19

Abstract:Scene text recognition is essential in many applications, including automated translation, information retrieval, driving assistance, and enhancing accessibility for individuals with visual impairments. Much research has been done to improve the accuracy and performance of scene text detection and recognition models. However, most of this research has been conducted in the most common languages, English and Chinese. There is a significant gap in low-resource languages, especially the Swahili Language. Swahili is widely spoken in East African countries but is still an under-explored language in scene text recognition. No studies have been focused explicitly on Swahili natural scene text detection and recognition, and no dataset for Swahili language scene text detection and recognition is publicly available. We propose a comprehensive dataset of Swahili scene text images and evaluate the dataset on different scene text detection and recognition models. The dataset contains 976 images collected in different places and under various circumstances. Each image has its annotation at the word level. The proposed dataset can also serve as a benchmark dataset specific to the Swahili language for evaluating and comparing different approaches and fostering future research endeavors. The dataset is available on GitHub via this link:

Computer Vision and Pattern Recognition
Text Recognition in the Wild: A Survey

Xiaoxue Chen,Lianwen Jin,Yuanzhi Zhu,Canjie Luo,Tianwei Wang

DOI: https://doi.org/10.48550/arXiv.2005.03492

2020-12-03

Abstract:The history of text can be traced back over thousands of years. Rich and precise semantic information carried by text is important in a wide range of vision-based application scenarios. Therefore, text recognition in natural scenes has been an active research field in computer vision and pattern recognition. In recent years, with the rise and development of deep learning, numerous methods have shown promising in terms of innovation, practicality, and efficiency. This paper aims to (1) summarize the fundamental problems and the state-of-the-art associated with scene text recognition; (2) introduce new insights and ideas; (3) provide a comprehensive review of publicly available resources; (4) point out directions for future work. In summary, this literature review attempts to present the entire picture of the field of scene text recognition. It provides a comprehensive reference for people entering this field, and could be helpful to inspire future research. Related resources are available at our Github repository: <a class="link-external link-https" href="https://github.com/HCIILAB/Scene-Text-Recognition" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
On Exploring and Improving Robustness of Scene Text Detection Models

Shilian Wu,Wei Zhai,Yongrui Li,Kewei Wang,Zengfu Wang

DOI: https://doi.org/10.48550/arXiv.2110.05700

2021-10-12

Abstract:It is crucial to understand the robustness of text detection models with regard to extensive corruptions, since scene text detection techniques have many practical applications. For systematically exploring this problem, we propose two datasets from which to evaluate scene text detection models: ICDAR2015-C (IC15-C) and CTW1500-C (CTW-C). Our study extends the investigation of the performance and robustness of the proposed region proposal, regression and segmentation-based scene text detection frameworks. Furthermore, we perform a robustness analysis of six key components: pre-training data, backbone, feature fusion module, multi-scale predictions, representation of text instances and loss function. Finally, we present a simple yet effective data-based method to destroy the smoothness of text regions by merging background and foreground, which can significantly increase the robustness of different text detection networks. We hope that this study will provide valid data points as well as experience for future research. Benchmark, code and data will be made available at \url{<a class="link-external link-https" href="https://github.com/wushilian/robust-scene-text-detection-benchmark" rel="external noopener nofollow">this https URL</a>}.

Computer Vision and Pattern Recognition
An Image Dataset of Text Patches in Everyday Scenes

Ahmed Ibrahim,A. Lynn Abbott,Mohamed E. Hussein

DOI: https://doi.org/10.48550/arXiv.1610.06494

2016-10-20

Computer Vision and Pattern Recognition

Abstract:This paper describes a dataset containing small images of text from everyday scenes. The purpose of the dataset is to support the development of new automated systems that can detect and analyze text. Although much research has been devoted to text detection and recognition in scanned documents, relatively little attention has been given to text detection in other types of images, such as photographs that are posted on social-media sites. This new dataset, known as COCO-Text-Patch, contains approximately 354,000 small images that are each labeled as "text" or "non-text". This dataset particularly addresses the problem of text verification, which is an essential stage in the end-to-end text detection and recognition pipeline. In order to evaluate the utility of this dataset, it has been used to train two deep convolution neural networks to distinguish text from non-text. One network is inspired by the GoogLeNet architecture, and the second one is based on CaffeNet. Accuracy levels of 90.2% and 90.9% were obtained using the two networks, respectively. All of the images, source code, and deep-learning trained models described in this paper will be publicly available
ShopSign: a Diverse Scene Text Dataset of Chinese Shop Signs in Street Views

Chongsheng Zhang,Guowen Peng,Yuefeng Tao,Feifei Fu,Wei Jiang,George Almpanidis,Ke Chen

DOI: https://doi.org/10.48550/arXiv.1903.10412

2019-03-25

Computer Vision and Pattern Recognition

Abstract:In this paper, we introduce the ShopSign dataset, which is a newly developed natural scene text dataset of Chinese shop signs in street views. Although a few scene text datasets are already publicly available (e.g. ICDAR2015, COCO-Text), there are few images in these datasets that contain Chinese texts/characters. Hence, we collect and annotate the ShopSign dataset to advance research in Chinese scene text detection and recognition. The new dataset has three distinctive characteristics: (1) large-scale: it contains 25,362 Chinese shop sign images, with a total number of 196,010 text-lines. (2) diversity: the images in ShopSign were captured in different scenes, from downtown to developing regions, using more than 50 different mobile phones. (3) difficulty: the dataset is very sparse and imbalanced. It also includes five categories of hard images (mirror, wooden, deformed, exposed and obscure). To illustrate the challenges in ShopSign, we run baseline experiments using state-of-the-art scene text detection methods (including CTPN, TextBoxes++ and EAST), and cross-dataset validation to compare their corresponding performance on the related datasets such as CTW, RCTW and ICPR 2018 MTWI challenge dataset. The sample images and detailed descriptions of our ShopSign dataset are publicly available at: https://github.com/chongshengzhang/shopsign.
Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognition

Chee Kheng Ch’ ng,Chee Seng Chan,Chee Kheng Ch'ng

DOI: https://doi.org/10.1109/icdar.2017.157

2017-11-01

Abstract:Text in curve orientation, despite being one of the common text orientations in real world environment, has close to zero existence in well received scene text datasets such as ICDAR{}^{\prime}13 and MSRA-TD500. The main motivation of Total-Text is to fill this gap and facilitate a new research direction for the scene text community. On top of conventional horizontal and multi-oriented text, it features curved-oriented text. Total-Text is highly diversified in orientations, more than half of its images have a combination of more than two orientations. Recently, a new breed of solutions that casted text detection as a segmentation problem has demonstrated their effectiveness against multi-oriented text. In order to evaluate its robustness against curved text, we fine-tuned DeconvNet and benchmark it on Total-Text. Total-Text with its annotation is available at https://github.com/cs-chan/Total-Text-Dataset.
Accurate Scene Text Detection Via Scale-Aware Data Augmentation and Shape Similarity Constraint

Pengwen Dai,Yang Li,Hua Zhang,Jingzhi Li,Xiaochun Cao

DOI: https://doi.org/10.1109/tmm.2021.3073575

IF: 7.3

2022-01-01

IEEE Transactions on Multimedia

Abstract:Scene text detection has attracted increasing concerns with the rapid development of deep neural networks in recent years. However, existing scene text detectors may overfit on the public datasets due to the limited training data, or generate inaccurate localization for arbitrary-shape scene texts. This paper presents an arbitrary-shape scene text detection method that can achieve better generalization ability and more accurate localization. We first propose a Scale-Aware Data Augmentation (SADA) technique to increase the diversity of training samples. SADA considers the scale variations and local visual variations of scene texts, which can effectively relieve the dilemma of limited training data. At the same time, SADA can enrich the training minibatch, which contributes to accelerating the training process. Furthermore, a Shape Similarity Constraint (SSC) technique is exploited to model the global shape structure of arbitrary-shape scene texts and backgrounds from the perspective of the loss function. SSC encourages the segmentation of text or non-text in the candidate boxes to be similar to the corresponding ground truth, which is helpful to localize more accurate boundaries for arbitrary-shape scene texts. Extensive experiments have demonstrated the effectiveness of the proposed techniques, and state-of-the-art performances are achieved over public arbitrary-shape scene text benchmarks (e.g., CTW1500, Total-Text and ArT).

computer science, information systems,telecommunications, software engineering
Scene Text Detection in Natural Images: A Review

Dongping Cao,Yong Zhong,Lishun Wang,Yilong He,Jiachen Dang

DOI: https://doi.org/10.3390/sym12121956

2020-11-26

Symmetry

Abstract:Scene text detection is attracting more and more attention and has become an important topic in machine vision research. With the development of mobile IoT (Internet of things) and deep learning technology, text detection research has made significant progress. This survey aims to summarize and analyze the main challenges and significant progress in scene text detection research. In this paper, we first introduce the history and progress of scene text detection and classify the traditional methods and deep learning-based methods in detail, pointing out the corresponding key issues and techniques. Then, we introduce commonly used benchmark datasets and evaluation protocols and identify state-of-the-art algorithms by comparison. Finally, we summarize and predict potential future research directions.
SketchyScene: Richly-Annotated Scene Sketches

Changqing Zou,Qian Yu,Ruofei Du,Haoran Mo,Yi-Zhe Song,Tao Xiang,Chengying Gao,Baoquan Chen,Hao Zhang

DOI: https://doi.org/10.48550/arXiv.1808.02473

2018-08-08

Abstract:We contribute the first large-scale dataset of scene sketches, SketchyScene, with the goal of advancing research on sketch understanding at both the object and scene level. The dataset is created through a novel and carefully designed crowdsourcing pipeline, enabling users to efficiently generate large quantities of realistic and diverse scene sketches. SketchyScene contains more than 29,000 scene-level sketches, 7,000+ pairs of scene templates and photos, and 11,000+ object sketches. All objects in the scene sketches have ground-truth semantic and instance masks. The dataset is also highly scalable and extensible, easily allowing augmenting and/or changing scene composition. We demonstrate the potential impact of SketchyScene by training new computational models for semantic segmentation of scene sketches and showing how the new dataset enables several applications including image retrieval, sketch colorization, editing, and captioning, etc. The dataset and code can be found at <a class="link-external link-https" href="https://github.com/SketchyScene/SketchyScene" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence
TextNeRF: A Novel Scene-Text Image Synthesis Method Based on Neural Radiance Fields

Jialei Cui,Jianwei Du,Wenzhuo Liu,Zhouhui Lian

DOI: https://doi.org/10.1109/cvpr52733.2024.02102

2024-01-01

Abstract:Acquiring large-scale, well-annotated datasets is essential for training robust scene text detectors, yet the process is often resource-intensive and time-consuming. While some efforts have been made to explore the synthesis of scene text images, a notable gap remains between syn-thetic and authentic data. In this paper, we introduce a novel method that utilizes Neural Radiance Fields (NeRF) to model real-world scenes and emulate the data collection process by rendering images from diverse camera per-spectives, enriching the variability and realism of the synthesized data. A semi-supervised learning framework is proposed to categorize semantic regions within 3D scenes, ensuring consistent labeling of text regions across various viewpoints. Our method also models the pose, and view-dependent appearance of text regions, thereby offering precise control over camera poses and significantly improving the realism of text insertion and editing within scenes. Employing our technique on real-world scenes has led to the creation of a novel scene text image dataset (https://github.com/cuijl-ai/TextNeRF). Compared to other existing benchmarks, the proposed dataset is distinctive in providing not only standard annotations such as bounding boxes and transcriptions but also the information of 3D pose attributes for text regions, enabling a more detailed evaluation of the robustness of text detection algorithms. Through extensive experiments, we demonstrate the effectiveness of our proposed method in enhancing the performance of scene text detectors.
Scene Parsing through ADE20K Dataset

Bolei Zhou,Hang Zhao,Xavier Puig,Sanja Fidler,Adela Barriuso,Antonio Torralba

DOI: https://doi.org/10.1109/cvpr.2017.544

2017-07-01

Abstract:Scene parsing, or recognizing and segmenting objects and stuff in an image, is one of the key problems in computer vision. Despite the community's efforts in data collection, there are still few image datasets covering a wide range of scenes and object categories with dense and detailed annotations for scene parsing. In this paper, we introduce and analyze the ADE20K dataset, spanning diverse annotations of scenes, objects, parts of objects, and in some cases even parts of parts. A scene parsing benchmark is built upon the ADE20K with 150 object and stuff classes included. Several segmentation baseline models are evaluated on the benchmark. A novel network design called Cascade Segmentation Module is proposed to parse a scene into stuff, objects, and object parts in a cascade and improve over the baselines. We further show that the trained scene parsing networks can lead to applications such as image content removal and scene synthesis11Dataset and pretrained models are available at http://groups.csail.mit.edu/vision/datasets/ADE20K/. Dataset and pretrained models are available at http://groups.csail.mit.edu/vision/datasets/ADE20K/
Stroke-Based Scene Text Erasing Using Synthetic Data for Training

Zhengmi Tang,Tomo Miyazaki,Yoshihiro Sugaya,Shinichiro Omachi

DOI: https://doi.org/10.48550/arXiv.2104.11493

2021-04-23

Computer Vision and Pattern Recognition

Abstract:Scene text erasing, which replaces text regions with reasonable content in natural images, has drawn significant attention in the computer vision community in recent years. There are two potential subtasks in scene text erasing: text detection and image inpainting. Both subtasks require considerable data to achieve better performance; however, the lack of a large-scale real-world scene-text removal dataset does not allow existing methods to realize their potential. To compensate for the lack of pairwise real-world data, we made considerable use of synthetic text after additional enhancement and subsequently trained our model only on the dataset generated by the improved synthetic text engine. Our proposed network contains a stroke mask prediction module and background inpainting module that can extract the text stroke as a relatively small hole from the cropped text image to maintain more background content for better inpainting results. This model can partially erase text instances in a scene image with a bounding box or work with an existing scene-text detector for automatic scene text erasing. The experimental results from the qualitative and quantitative evaluation on the SCUT-Syn, ICDAR2013, and SCUT-EnsText datasets demonstrate that our method significantly outperforms existing state-of-the-art methods even when they are trained on real-world data.
Scene text detection and recognition: a survey

Fatemeh Naiemi,Vahid Ghods,Hassan Khalesi

DOI: https://doi.org/10.1007/s11042-022-12693-7

IF: 2.577

2022-03-11

Multimedia Tools and Applications

Abstract:Scene text detection and recognition have been given a lot of attention in recent years and have been used in many vision-based applications. In this field, there are various types of challenges, including images with wavy text, images with text rotation and orientation, changing the scale and variety of text fonts, noisy images, wild background images, which make the detection and recognition of text from the image more complex and difficult. In this article, we first presented a comprehensive review of recent advances in text detection and recognition and described the advantages and disadvantages. The common datasets were introduced. Then, the recent methods compared together and analyzed the text detection and recognition systems. According to the recent decade studies, one of the most important challenges is curved and vertical text detection in this field. We have expressed approaches for the development of the detection and recognition system. Also, we have described the methods that are robust in the detection and recognition of curved and vertical texts. Finally, we have presented some approaches to develop text detection and recognition systems as the future work.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
Robust Text Detection in Natural Scene Images

Xu-Cheng Yin,Xuwang Yin,Kaizhu Huang,Hong-Wei Hao

DOI: https://doi.org/10.1109/TPAMI.2013.182

2013-06-03

Abstract:Text detection in natural scene images is an important prerequisite for many content-based image analysis tasks. In this paper, we propose an accurate and robust method for detecting texts in natural scene images. A fast and effective pruning algorithm is designed to extract Maximally Stable Extremal Regions (MSERs) as character candidates using the strategy of minimizing regularized variations. Character candidates are grouped into text candidates by the ingle-link clustering algorithm, where distance weights and threshold of the clustering algorithm are learned automatically by a novel self-training distance metric learning algorithm. The posterior probabilities of text candidates corresponding to non-text are estimated with an character classifier; text candidates with high probabilities are then eliminated and finally texts are identified with a text classifier. The proposed system is evaluated on the ICDAR 2011 Robust Reading Competition dataset; the f measure is over 76% and is significantly better than the state-of-the-art performance of 71%. Experimental results on a publicly available multilingual dataset also show that our proposed method can outperform the other competitive method with the f measure increase of over 9 percent. Finally, we have setup an online demo of our proposed scene text detection system at <a class="link-external link-http" href="http://kems.ustb.edu.cn/learning/yin/dtext" rel="external noopener nofollow">this http URL</a>.

Computer Vision and Pattern Recognition,Information Retrieval,Machine Learning
StacMR: Scene-Text Aware Cross-Modal Retrieval

Andrés Mafla,Rafael S. Rezende,Lluís Gómez,Diane Larlus,Dimosthenis Karatzas,Andres Mafla,Lluis Gomez

DOI: https://doi.org/10.1109/wacv48630.2021.00227

2021-01-01

Abstract:Recent models for cross-modal retrieval have benefited from an increasingly rich understanding of visual scenes, afforded by scene graphs and object interactions to mention a few. This has resulted in an improved matching between the visual representation of an image and the textual representation of its caption. Yet, current visual representations overlook a key aspect: the text appearing in images, which may contain crucial information for retrieval. In this paper, we first propose a new dataset that allows exploration of cross-modal retrieval where images contain scene-text instances. Then, armed with this dataset, we describe several approaches which leverage scene text, including a better scene-text aware cross-modal retrieval method which uses specialized representations for text from the captions and text from the visual scene, and reconcile them in a common embedding space. Extensive experiments confirm that cross-modal retrieval approaches benefit from scene text and highlight interesting research questions worth exploring further. Dataset and code are available at europe.naverlabs.com/stacmr.
KhmerST: A Low-Resource Khmer Scene Text Detection and Recognition Benchmark

Vannkinh Nom,Souhail Bakkali,Muhammad Muzzamil Luqman,Mickaël Coustaty,Jean-Marc Ogier

2024-10-24

Abstract:Developing effective scene text detection and recognition models hinges on extensive training data, which can be both laborious and costly to obtain, especially for low-resourced languages. Conventional methods tailored for Latin characters often falter with non-Latin scripts due to challenges like character stacking, diacritics, and variable character widths without clear word boundaries. In this paper, we introduce the first Khmer scene-text dataset, featuring 1,544 expert-annotated images, including 997 indoor and 547 outdoor scenes. This diverse dataset includes flat text, raised text, poorly illuminated text, distant and partially obscured text. Annotations provide line-level text and polygonal bounding box coordinates for each scene. The benchmark includes baseline models for scene-text detection and recognition tasks, providing a robust starting point for future research endeavors. The KhmerST dataset is publicly accessible at <a class="link-external link-https" href="https://gitlab.com/vannkinhnom123/khmerst" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

Advances of Scene Text Datasets

Scene text detection and recognition with advances in deep learning: a survey

Scene text detection and recognition: recent advances and future trends

Scene Text Detection and Recognition: The Deep Learning Era

The First Swahili Language Scene Text Detection and Recognition Dataset

Text Recognition in the Wild: A Survey

On Exploring and Improving Robustness of Scene Text Detection Models

An Image Dataset of Text Patches in Everyday Scenes

ShopSign: a Diverse Scene Text Dataset of Chinese Shop Signs in Street Views

Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognition

Accurate Scene Text Detection Via Scale-Aware Data Augmentation and Shape Similarity Constraint

Scene Text Detection in Natural Images: A Review

SketchyScene: Richly-Annotated Scene Sketches

TextNeRF: A Novel Scene-Text Image Synthesis Method Based on Neural Radiance Fields

Scene Parsing through ADE20K Dataset

Stroke-Based Scene Text Erasing Using Synthetic Data for Training

Scene text detection and recognition: a survey

Robust Text Detection in Natural Scene Images

StacMR: Scene-Text Aware Cross-Modal Retrieval

KhmerST: A Low-Resource Khmer Scene Text Detection and Recognition Benchmark