Abstract:A novel music/speech analysis framework via integration of audio content and functional brain response.Brain response features are derived from fMRI data when participants are listening to the music/speech.A novel regression algorithm named ITGP is proposed to improve the quality in predicting functional brain response. Effective analysis of music/speech data such as clustering, retrieval, and classification has received significant attention in recent years. Traditional methods mainly rely on the low-level acoustic features derived from digital audio stream, and the accuracy of these methods is limited by the well-known semantic gap. To alleviate this problem, we propose a novel framework for music/speech clustering, retrieval, and classification by integrating the low-level acoustic features derived from audio content with the functional magnetic resonance imaging (fMRI) measured features that represent the brain's functional response when subjects are listening to the music/speech excerpts. First, the brain networks and regions of interest (ROIs) involved in the comprehension of audio stimuli, such as the auditory, emotion, attention, and working memory systems, are located by a new approach named dense individualized and common connectivity-based cortical landmarks (DICCCOLs). Then the functional connectivity matrix measuring the similarity between the fMRI signals of different ROIs is adopted to represent the brain's comprehension of audio semantics. Afterwards, we propose an improved twin Gaussian process (ITGP) model based on self-training to predict the fMRI-measured features of testing data without fMRI scanning. Finally, multi-view learning algorithms are proposed to integrate acoustic features with fMRI-measured features for music/speech clustering, retrieval, and classification, respectively. The experimental results demonstrate the superiority of our proposed work in comparison with existing methods and suggest the advantage of integrating functional brain responses via fMRI data for music/speech analysis.

Audio Analysis Approach based on General Model

A Method Based On General Model Template For Audio Retrieval

Hybrid Support Vector Machine and General Model Approach for Audio Classification

A Method Based on General Model and Rough Set for Audio Classification

Research on methodology of image semantic understanding based on generalized computing

An Approach Based on Reverse Hidden Markov Model for Audio Classification

Multiple Kernel Visual-Auditory Representation Learning for Retrieval

Audio Sentiment Analysis by Heterogeneous Signal Features Learned from Utterance-Based Parallel Neural Network.

Music Waveform Analysis Based on SOM Neural Network and Big Data

An Overview of Semantics Processing in Content-Based 3D Model Retrieval

Hybrid Svm/Hmm Approach For Audio Classification

Hybrid Independent Component Analysis And Support Vector Machine Approach For Audio Classification

Audio Similarity Measure by Graph Modeling and Matching

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Self- Motion Analysis of Audio Music Based on Content and Searching Technology Research

Hybrid Independent Component Analysis and Rough Set Approach for Audio Feature Extraction

Hierarchical Support Vector Machines for Audio Classification

Audio scene semantic similarity computing approach

Audio Retrieval: Based On Unsupervised Learning Approach

Semantic Concept Annotation Based on Audio PLSA Model

Analysis of music/speech via integration of audio content and functional brain response.