Abstract:The topics of visual and audio quality assessment (QA) have been widely researched for decades, yet nearly all of this prior work has focused only on single-mode visual or audio signals. However, visual signals rarely are presented without accompanying audio, including heavy-bandwidth video streaming applications. Moreover, the distortions that may separately (or conjointly) afflict the visual and audio signals collectively shape user-perceived quality of experience (QoE). This motivated us to conduct a subjective study of audio and video (A/V) quality, which we then used to compare and develop A/V quality measurement models and algorithms. The new LIVE-SJTU Audio and Video Quality Assessment (A/V-QA) Database includes 336 A/V sequences that were generated from 14 original source contents by applying 24 different A/V distortion combinations on them. We then conducted a subjective A/V quality perception study on the database towards attaining a better understanding of how humans perceive the overall combined quality of A/V signals. We also designed four different families of objective A/V quality prediction models, using a multimodal fusion strategy. The different types of A/V quality models differ in both the unimodal audio and video quality prediction models comprising the direct signal measurements and in the way that the two perceptual signal modes are combined. The objective models are built using both existing state-of-the-art audio and video quality prediction models and some new prediction models, as well as quality-predictive features delivered by a deep neural network. The methods of fusing audio and video quality predictions that are considered include simple product combinations as well as learned mappings. Using the new subjective A/V database as a tool, we validated and tested all of the objective A/V quality prediction models. We will make the database publicly available to facilitate further research.

Quantitative Analysis of Audio-Visual Tasks: An Information-Theoretic Perspective

Quantitative similarity computing for audio effect semantic in video content analysis

Exploring Effective Relationships Between Visual-Audio Channels in Data Visualization

Audio-based Digital Multimedia Content Analysis and Its Visualization

QoE Evaluation of Multimedia Services Based on Audiovisual Quality and User Interest.

Understanding the Quality of User Experience in Telepresence Systems from an Information Theory Perspective

Distinct Mechanism of Audiovisual Integration with Informative and Uninformative Sound in a Visual Detection Task: A DCM Study

Study of Subjective and Objective Quality Assessment of Audio-Visual Signals.

A Survey of Vision and Language Related Multi-Modal Task

Audio-Visual Speaker Tracking: Progress, Challenges, and Future Directions

A Survey on Audio Synthesis and Audio-Visual Multimodal Processing

Sound Influences Visual Attention Discriminately in Videos

Learning in Audio-visual Context: A Review, Analysis, and New Perspective

A Study of Speech Intention Understanding Based on Multimodal Information Integration

A Comparative Study of Perceptual Quality Metrics for Audio-driven Talking Head Videos

A Variety Of Visual-Speech Matching Erp Studies In Quiet-Noise Scenarios

Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention

An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation

Coherence Research of Audio-Visual Cross-Modal Based on HHT.

Uncovering the Visual Contribution in Audio-Visual Speech Recognition

Exploration of Audio Quality Assessment and Anomaly Localisation Using Attention Models