Abstract:Lip‐reading deciphers speech without audio data, and deep learning advancements have improved lip‐reading in English and Chinese. Cantonese lip‐reading sentences, a Cantonese lip‐reading dataset, and a novel visual frontend, 3D‐visual attention net, which achieves comparable performance on Chinese Mandarin lip reading dataset, lip reading sentences 2, lip reading sentences 3, and Cantonese lip‐reading sentences datasets, are introduced. This addresses the scarcity of Cantonese research and provides a new foundation for dialect lip‐reading, fostering the advancement of Cantonese lip‐reading tasks. Lip‐reading deciphers speech by observing lip movements without relying on audio data. The rapid advancements in deep learning have significantly improved lip‐reading for both English and Chinese; however, research on dialects such as Cantonese remains scarce. Consequently, most Chinese lip‐reading datasets focus on Mandarin, with only a few addressing Cantonese. To bridge this gap, a sentence‐level Cantonese lip‐reading dataset, designated as Cantonese lip‐reading sentences are introduced, comprising over 500 unique speakers and more than 30,000 samples. To ensure alignment with real‐world scenarios, no restrictions are imposed on factors such as gender, age, posture, lighting conditions, or speech rate. A comprehensive description of the pipeline employed is provided for collecting and constructing the dataset and introduce an innovative visual frontend, 3D‐visual attention net. This frontend combines the advantages of convolution and self‐attention mechanisms to extract fine‐grained lip region features. These features are subsequently input into the conformer backend for temporal sequence modelling, achieving comparable performance on Chinese Mandarin lip reading dataset, lip reading sentences 2, lip reading sentences 3, and Cantonese lip‐reading sentences datasets. Benchmark tests on Cantonese lip‐reading sentences demonstrate the challenges it poses, providing a novel research foundation for dialect lip‐reading and fostering the advancement of Cantonese lip‐reading tasks.

Automatic Speech Recognition Datasets in Cantonese: A Survey and New Dataset

CI-AVSR: A Cantonese Audio-Visual Speech Dataset for In-car Command Recognition

Kaggle Competition: Cantonese Audio-Visual Speech Recognition for In-car Commands

Automatic Collecting of Text Data for Cantonese Language Modeling

Cantonese neural speech synthesis from found newscasting video data and its speaker adaptation

Advances in Cantonese Speech Recognition: A Language-Specific Pretraining Model and RNN-T Loss

Recent Progress in the CUHK Dysarthric Speech Recognition System

Cantonese sentence dataset for lip‐reading

Speech recognition datasets for low-resource Congolese languages

Cantonese natural language processing in the transformers era: a survey and current challenges

ChildMandarin: A Comprehensive Mandarin Speech Dataset for Young Children Aged 3-5

TALCS: An Open-Source Mandarin-English Code-Switching Corpus and a Speech Recognition Baseline

MAC: A unified framework boosting low resource automatic speech recognition

Cantonese Automatic Speech Recognition Using Transfer Learning from Mandarin

A CANTONESE ACCENT CHINESE SPEECH CORPUS

M2ASR-MONGO: A Free Mongolian Speech Database and Accompanied Baselines

A reweighting method for speech recognition with imbalanced data of Mandarin and sub-dialects

KeSpeech: an Open Source Speech Dataset of Mandarin and Its Eight Subdialects.

OC16-CE80: A Chinese-English Mixlingual Database and A Speech Recognition Baseline

Real and synthetic Punjabi speech datasets for automatic speech recognition

ATCSpeech: A Multilingual Pilot-Controller Speech Corpus from Real Air Traffic Control Environment