Abstract:An increasing number of Chinese people are troubled by different degrees of visual impairment, which has made the modal conversion between a single image or video frame in the visual field and the audio expressing the same information a research hotspot. Deep learning technologies such as OCR+Vocoder and Im2Wav enable English audio synthesis or image-to-sound matching in a self-supervised manner. However, the audio data used for training is limited and English is not universal for visually impaired people with different educational levels. Therefore, for the sake of solving the problems of data volume and language applicability to improve the reading efficiency of visually impaired people, a set of image-to-speech framework CLIP-KNN-Fastspeech2 based on the Chinese context was constructed. The framework integrates multiple basic models and adopts the strategy of independent pre-training and joint fine-tuning. First, the Chinese CLIP and Fastspeech2 text-to-speech models were pre-trained on two public datasets, MUGE and Baker, respectively, and their convergence was verified. Subsequently, joint fine-tuning was performed using a self-built Braille image dataset. Experimental results on multiple public datasets such as VGGSound, Flickr8k, ImageHear, and the self-built Braille dataset BIT-DP show that the model has improved objective indicators such as BLEU4,FAD(Fréchet Audio Distance), WER(Word Error Ratio), and even inference speed. This verifies that the constructed model still has the ability to synthesize high-quality speech under limited data, and also proves the effectiveness of the joint training strategy that integrates multiple basic models.

Bridging Language Gaps in Audio-Text Retrieval

AudioVSR: Enhancing Video Speech Recognition with Audio Data

Audio Retrieval with WavText5K and CLAP Training

Language-based Audio Retrieval with GPT-Augmented Captions and Self-Attended Audio Clips

Audio-Text Retrieval in Context

Language-based Audio Retrieval with Co-Attention Networks

Audio Retrieval with Natural Language Queries: A Benchmark Study

Retrieval-Augmented Text-to-Audio Generation

Contrastive Latent Space Reconstruction Learning for Audio-Text Retrieval

Braille-to-Speech Generator: Audio Generation Based on Joint Fine-Tuning of CLIP and Fastspeech2

Looking and Listening: Audio Guided Text Recognition

Audio–text retrieval based on contrastive learning and collaborative attention mechanism

Language-Queried Target Sound Extraction Without Parallel Training Data

Matching Text and Audio Embeddings: Exploring Transfer-learning Strategies for Language-based Audio Retrieval

CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models

Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding

Multi-grained Representation Learning for Cross-modal Retrieval

Advancing Natural-Language Based Audio Retrieval with PaSST and Large Audio-Caption Data Sets

Text-based Audio Retrieval by Learning from Similarities between Audio Captions

Hear the Scene: Audio-Enhanced Text Spotting

Exploring the Role of Audio in Video Captioning