Abstract:Determining the readability of a text is the first step to its simplification. In this paper, we present a readability analysis tool capable of analyzing text written in the Bengali language to provide in-depth information on its readability and complexity. Despite being the 7th most spoken language in the world with 230 million native speakers, Bengali suffers from a lack of fundamental resources for natural language processing. Readability related research of the Bengali language so far can be considered to be narrow and sometimes faulty due to the lack of resources. Therefore, we correctly adopt document-level readability formulas traditionally used for U.S. based education system to the Bengali language with a proper age-to-age comparison. Due to the unavailability of large-scale human-annotated corpora, we further divide the document-level task into sentence-level and experiment with neural architectures, which will serve as a baseline for the future works of Bengali readability prediction. During the process, we present several human-annotated corpora and dictionaries such as a document-level dataset comprising 618 documents with 12 different grade levels, a large-scale sentence-level dataset comprising more than 96K sentences with simple and complex labels, a consonant conjunct count algorithm and a corpus of 341 words to validate the effectiveness of the algorithm, a list of 3,396 easy words, and an updated pronunciation dictionary with more than 67K words. These resources can be useful for several other tasks of this low-resource language. We make our Code & Dataset publicly available at <a class="link-external link-https" href="https://github.com/tafseer-nayeem/BengaliReadability" rel="external noopener nofollow">this https URL</a>} for reproduciblity.

Metrics for Bengali Text Entry Research

Simple or Complex? Learning to Predict Readability of Bengali Texts

Enhancement of Bengali OCR by Specialized Models and Advanced Techniques for Diverse Document Types

An Automated Evaluation Metric for Chinese Text Entry

Opinion Mining of Bengali Review Written with English Character Using Machine Learning Approaches

Classification Benchmarks for Under-resourced Bengali Language based on Multichannel Convolutional-LSTM Network

Authorship Identification in Bengali Literature: a Comparative Analysis

Challenges and Opportunities of Speech Recognition for Bengali Language

Smart Bengali Cell Phone Keypad Layout

Automatic Speech Recognition for Biomedical Data in Bengali Language

Generation of simulated data for Bengali text localization in natural images

Bengali Slang detection using state-of-the-art supervised models from a given text

Bengali & Banglish: A monolingual dataset for emotion detection in linguistically diverse contexts

Byte Pair Encoding Is All You Need For Automatic Bengali Speech Recognition

Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for Bengali-English Machine Translation

Sentiment analysis in multilingual context: Comparative analysis of machine learning and hybrid deep learning models

Enhancing Sentiment Analysis in Bengali Texts: A Hybrid Approach Using Lexicon-Based Algorithm and Pretrained Language Model Bangla-BERT

Bangla Natural Language Processing: A Comprehensive Analysis of Classical, Machine Learning, and Deep Learning-Based Methods

BN-AuthProf: Benchmarking Machine Learning for Bangla Author Profiling on Social Media Texts

Preparing Bengali-English Code-Mixed Corpus for Sentiment Analysis of Indian Languages

Inter-Rater Agreement Study on Readability Assessment in Bengali