Abstract:The topics of visual and audio quality assessment (QA) have been widely researched for decades, yet nearly all of this prior work has focused only on single-mode visual or audio signals. However, visual signals rarely are presented without accompanying audio, including heavy-bandwidth video streaming applications. Moreover, the distortions that may separately (or conjointly) afflict the visual and audio signals collectively shape user-perceived quality of experience (QoE). This motivated us to conduct a subjective study of audio and video (A/V) quality, which we then used to compare and develop A/V quality measurement models and algorithms. The new LIVE-SJTU Audio and Video Quality Assessment (A/V-QA) Database includes 336 A/V sequences that were generated from 14 original source contents by applying 24 different A/V distortion combinations on them. We then conducted a subjective A/V quality perception study on the database towards attaining a better understanding of how humans perceive the overall combined quality of A/V signals. We also designed four different families of objective A/V quality prediction models, using a multimodal fusion strategy. The different types of A/V quality models differ in both the unimodal audio and video quality prediction models comprising the direct signal measurements and in the way that the two perceptual signal modes are combined. The objective models are built using both existing state-of-the-art audio and video quality prediction models and some new prediction models, as well as quality-predictive features delivered by a deep neural network. The methods of fusing audio and video quality predictions that are considered include simple product combinations as well as learned mappings. Using the new subjective A/V database as a tool, we validated and tested all of the objective A/V quality prediction models. We will make the database publicly available to facilitate further research.

Measuring Human Perception of Audiovisual Errors Using EEG

Neural Tracking of Sound Rhythms Correlates with Diagnosis, Severity, and Prognosis of Disorders of Consciousness

Calibrating Human Perception Threshold of Video Distortion Using EEG.

A Method to Study Global Spatial Patterns Related to Sensory Perception in Scalp EEG.

An EEG-Based Study on Perception of Video Distortion under Various Content Motion Conditions

PEAVS: Perceptual Evaluation of Audio-Visual Synchrony Grounded in Viewers' Opinion Scores

Brain-Inspired Image Perceptual Quality Assessment Based on EEG: A QoE Perspective

A subjective study of the perceptual acceptability of audio-video desynchronization in sports videos

Study on synchronization of ERP evoked by simultaneous visual-audio stimuli

Video Quality Measurement for Buffering Time Based on EEG Frequency Feature

A Method for Real-time Detection of Binocular Rivalry through Electroencephalogram

A Novel EEG Based Directed Transfer Function for Investigating Human Perception to Audio Noise.

Single-trial detection of EEG error-related potentials in serial visual presentation paradigm

On Attention Modules for Audio-Visual Synchronization

Human Perception Measurement by Electroencephalography for Facial Image Compression.

Study of Subjective and Objective Quality Assessment of Audio-Visual Signals.

EEG-based Objective Quality Assessment of Interrupted Speech

Biological Computation Indexes of Brain Oscillations in Unattended Facial Expression Processing Based on Event-Related Synchronization/Desynchronization

Asynchronous Video Target Detection Based on Single-Trial EEG Signals

Inter-Region Synchronization Analysis Based On Heterogeneous Matrix Similarity Measurement

Visual Attention Measurement Based on Electroencephalogram Feature Learning