Abstract:Automatic audio content recognition has attracted an increasing attention for developing multimedia systems, for which the most popular approaches combine frame-based features with statistic models or discriminative classifiers. The existing methods are effective for clean single-source event detection but may not perform well for unstructured environmental sounds, which have a broad noise-like flat spectrum and a diverse variety of compositions. We present an automatic acoustic scene understanding framework that detects audio events through two hierarchies, acoustic scene recognition and audio event recognition , in which the former is preceded by following dominant audio sources and in turn helps infer non-dominant audio events within the same scene through modeling their occurrence correlations. On the scene recognition hierarchy, we perform adaptive segmentation and feature extraction for every input acoustic scene stream through Eigen-audiospace and an optimized feature subspace, respectively. After filtering background, scene streams are recognized by modeling the observation density of dominant features using a two-level hidden Markov model. On the audio event recognition hierarchy, scene knowledge is characterized by an audio context model that essentially describes the occurrence correlations of dominant and non-dominant audio events within this scene. Monte Carlo integration and gradient descent techniques are employed to maximize the likelihood and correctly tag each audio event. To the best of our knowledge, this is the first work that models event correlations as scene context for robust audio event detection from complex and noisy environments. Note that according to the recent report, the mean accuracy for the acoustic scene classification task by human listeners is only around 71 % on the data collected in office environments from the DCASE dataset. None of the existing methods performs well on all scene categories and the average accuracy of the best performances of the recent 11 methods is 53.8 %. The proposed method averagely achieves an accuracy of 62.3 % on the same dataset. Additionally, we create a 10-CASE dataset by manually collecting 5,250 audio clips of 10 scene types and 21 event categories. Our experimental results on 10-CASE show that the proposed method averagely achieves the enhanced performance of 78.3 %, and the average accuracy of audio event recognition can be effectively improved by capturing dominant audio sources and reasoning non-dominant events from the dominant ones through acoustic context modeling. In the future work, exploring the interactions between acoustic scene recognition and audio event detection, and incorporating other modalities to improve the accuracy are required to further advance the proposed framework.

Salient environmental sound detection framework for machine awareness.

Computational Modeling of Environment Deviant Sound Detection Based on Human Auditory Cognitive Mechanism

Robust Audio Sensing with Multi-Sound Classification.

Active Object Discovery and Localization Using Sound-Induced Attention

Smart Ambient Sound Analysis via Structured Statistical Modeling.

Human–machine collaboration based sound event detection

Context-based environmental audio event recognition for scene understanding

A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!

Are acoustics enough? Semantic effects on auditory salience in natural scenes

Detection, classification and localization of acoustic events in the presence of background noise for acoustic surveillance of hazardous situations

Decoding Auditory Saliency From Fmri Brain Imaging

Environmental Sound Classification Framework Based on L-mHP Features and SE-ResNet50 Network Model

Interpretable and Robust Machine Learning for Exploring and Classifying Soundscape Data

Audiovisual Saliency Prediction Via Deep Learning

Novel sound event and sound activity detection framework based on intrinsic mode functions and deep learning

Improving Sound Event Localization and Detection with Class-Dependent Sound Separation for Real-World Scenarios

Salient Detection Via the Fusion of Background-Based and Multiscale Frequency-Domain Features

Large Scale Environmental Sound Classification Based on Efficient Feature Extraction.

Environmental Sound Recognition Using Double-Level Energy Detection

Sound Event Localization and Detection Using Parallel Multi-attention Enhancement

Artificial intelligence-based collaborative acoustic scene and event classification to support urban soundscape analysis and classification