Abstract:Asynchronization issue caused by different types of modalities is one of the major problems in audio visual speech recognition (AVSR) research. However, most AVSR systems merely rely on up sampling of video or down sampling of audio to align audio and visual features, assuming that the feature sequences are aligned frame-by-frame. These pre-processing steps oversimplify the asynchrony relation between acoustic signal and lip motion, lacking flexibility and impairing the performance of the system. Although there are systems modeling the asynchrony between the modalities, sometimes they fail to align speech and video precisely over some even all noisy conditions. In this paper, we propose a mutual feature alignment method for AVSR which can make full use of cross modility information to address the asynchronization issue by introducing Mutual Iterative Attention (MIA) mechanism. Our method can automatically learn an alignment in a mutual way by performing mutual attention iteratively between the audio and visual features, relying on the modified encoder structure of Transformer. Experimental results show that our proposed method obtains absolute improvements up to 20.42% over the audio modality alone depending upon the signal-to-noise-ratio (SNR) level. Better recognition performance can also be achieved comparing with the traditional feature concatenation method under both clean and noisy conditions. It is expectable that our proposed mutual feature alignment method can be easily generalized to other multimodal tasks with semantically correlated information.

Improve Audio Representation by Using Feature Structure Patterns

Audio Similarity Measure by Graph Modeling and Matching

Exploring composite acoustic features for efficient music similarity query.

Daubechies Wavelets Based Robust Audio Fingerprinting for Content-Based Audio Retrieval

Understanding Audio Pattern Using Convolutional Neural Network From Raw Waveforms

Improvement on Automatic Speech Segmentation Using Wavelet Packet Transform Features

Enhance Popular Music Emotion Regression by Importing Structure Information

Hybrid Independent Component Analysis and Rough Set Approach for Audio Feature Extraction

Acoustical feature analysis and optimization for aesthetic recognition of Chinese traditional music

Fractal Dimension Pattern Based Multiresolution Analysis for Rough Estimator of Person-Dependent Audio Emotion Recognition

Perceptual Similarity Between Audio Clips and Feature Selection for Its Measurement

Mutual Alignment between Audiovisual Features for End-to-End Audiovisual Speech Recognition

An Overview on Sound Features in Time and Frequency Domain

A Method Based on General Model and Rough Set for Audio Classification

A hybrid scheme for encoding audio signal using hidden Markov models of waveforms

A Method Based On General Model Template For Audio Retrieval

Speech Enhancement Based On Analysis Synthesis Framework With Improved Pitch Estimation And Spectral Envelope Enhancement

Improved Audio Embeddings by Adjacency-Based Clustering with Applications in Spoken Term Detection

A Compact and Discriminative Feature Based on Auditory Summary Statistics for Acoustic Scene Classification

Speech Enhancement Using Harmonics Regeneration Based on Multiband Excitation

The research on speech enhancement based on the simulation of auditory model using frame-synchronized combined wavelet packet transform algorithms