Abstract:An increasing number of Chinese people are troubled by different degrees of visual impairment, which has made the modal conversion between a single image or video frame in the visual field and the audio expressing the same information a research hotspot. Deep learning technologies such as OCR+Vocoder and Im2Wav enable English audio synthesis or image-to-sound matching in a self-supervised manner. However, the audio data used for training is limited and English is not universal for visually impaired people with different educational levels. Therefore, for the sake of solving the problems of data volume and language applicability to improve the reading efficiency of visually impaired people, a set of image-to-speech framework CLIP-KNN-Fastspeech2 based on the Chinese context was constructed. The framework integrates multiple basic models and adopts the strategy of independent pre-training and joint fine-tuning. First, the Chinese CLIP and Fastspeech2 text-to-speech models were pre-trained on two public datasets, MUGE and Baker, respectively, and their convergence was verified. Subsequently, joint fine-tuning was performed using a self-built Braille image dataset. Experimental results on multiple public datasets such as VGGSound, Flickr8k, ImageHear, and the self-built Braille dataset BIT-DP show that the model has improved objective indicators such as BLEU4,FAD(Fréchet Audio Distance), WER(Word Error Ratio), and even inference speed. This verifies that the constructed model still has the ability to synthesize high-quality speech under limited data, and also proves the effectiveness of the joint training strategy that integrates multiple basic models.

Bi-level Codebook Based Speech-driven Visual-speech Synthesis System

Learning based visual speech synthesis system

Deep Audio-visual System for Closed-set Word-level Speech Recognition

Visual Speech Synthesis Based on Learning Model

A Survey on Speech-synch Visual Speech Synthesizing Techniques

Text-To-Visual Speech in Chinese Based on Data-Driven Approach

Realistic Visual Speech Synthesis Based on Hybrid Concatenation Method

Lip Movement Synthesis in Audio-Visual Speech Recognition System

VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion

A Codebook-Drivfn Speech Enhancement Method By Exploiting Speech Harmonicity

Visual Speech Synthesis Algorithm Based on Chinese Visual Triphone

Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling

Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation

Improved Codebook-Based Speech Enhancement Based On Mbe Model

Super Intelligent Visual Auditory Information Processing System

VISUAL SPEECH SYNTHESIS BASED ON ARTICULATORY TRAJECTORY

GA-Based Speaking Mouth Correlative Speech Feature Abstraction

Braille-to-Speech Generator: Audio Generation Based on Joint Fine-Tuning of CLIP and Fastspeech2

The Study Of Computer-Aided Speech Training Method For Deaf Children Based On Learning Vector Quantization

ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders

Optimal Codebook Design Based on Genetic Algorithm