Deep Belief Networks Based Voice Activity Detection

ZhangXiao-Lei,WuJi

2013-01-01

IEEE Transactions on Audio, Speech, and Language Processing

Abstract:Fusing the advantages of multiple acoustic features is important for the robustness of voice activity detection (VAD). Recently, the machine-learning-based VADs have shown a superiority to traditio...

What problem does this paper attempt to address?

Deep Belief Networks Based Voice Activity Detection

Xiao-Lei Zhang,Ji Wu

DOI: https://doi.org/10.1109/TASL.2012.2229986

2013-01-01

IEEE Transactions on Audio, Speech, and Language Processing

Abstract:Fusing the advantages of multiple acoustic features is important for the robustness of voice activity detection (VAD). Recently, the machine-learning-based VADs have shown a superiority to traditional VADs on multiple feature fusion tasks. However, existing machine-learning-based VADs only utilize shallow models, which cannot explore the underlying manifold of the features. In this paper, we propose to fuse multiple features via a deep model, called deep belief network (DBN). DBN is a powerful hierarchical generative model for feature extraction. It can describe highly variant functions and discover the manifold of the features. We take the multiple serially-concatenated features as the input layer of DBN, and then extract a new feature by transferring these features through multiple nonlinear hidden layers. Finally, we predict the class of the new feature by a linear classifier. We further analyze that even a single-hidden-layer-based belief network is as powerful as the state-of-the-art models in the machine-learning-based VADs. In our empirical comparison, ten common features are used for performance analysis. Extensive experimental results on the AURORA2 corpus show that the DBN-based VAD not only outperforms eleven referenced VADs, but also can meet the real-time detection demand of VAD. The results also show that the DBN-based VAD can fuse the advantages of multiple features effectively.
A Novel and Efficient Voice Activity Detector Using Shape Features of Speech Wave.

Qiming Zhao,Yingchun Yang,Hong Li

DOI: https://doi.org/10.1007/978-3-319-12484-1_42

2014-01-01

Abstract:A voice activity detector (VAD) is the prerequisite for speaker recognition in real life. Currently, we deal with the VAD problem at the frame level through short time window function. However, when tackling with the VAD problem manually, we can easily pick out the speech segments containing several words. Inspired by this, we firstly use IIR filter to get the envelope of the waveform and divide the envelope into separate sound segments. And then we extract shape features from the obtained segments and use K-means to cluster the data featured by the amplitude of the wave crest to discard the silent part. Finally, we utilize other shape features to discard the noise part. The performance of our proposed VAD method has apparently surpassed the energy-based VAD and VQVAD with a relative 20% decrease in error rate, While the computation time of the proposed VAD method is only 30% less than that of VQVAD. We also get an encouraging result utilizing our VAD method for speaker recognition with about 3% average decrease in EER.
Applying Support Vector Machines to Voice Activity Detection

EQ Dong,GZ Liu,YT Zhou,XD Zhang

DOI: https://doi.org/10.1109/icosp.2002.1179987

2003-01-01

Abstract:A new voice activity detector (VAD) algorithm using support vector machines (SVM) is proposed in the paper, and the new VAD effectiveness is validated. The sequential minimal optimization (SMO) algorithm for fast training support vector machines is adopted. The proposed VAD algorithm via SVM (SVM-VAD) also uses the characteristic parameters set used by G.729 Annex B (G.729B) VAD. Comparing SVM-VAD with G729B VAD shows that it is effective for applying SVM to VAD. The new proposed VAD algorithm is integrated with G.729B instead of G.729B VAD, informal listening tests show that the integrated speech coding system has a little better efficiency over the G.729B VAD in perceptivity.
Denoising Deep Neural Networks Based Voice Activity Detection

Xiao-Lei Zhang,Ji Wu

DOI: https://doi.org/10.1109/ICASSP.2013.6637769

2013-03-04

Abstract:Recently, the deep-belief-networks (DBN) based voice activity detection (VAD) has been proposed. It is powerful in fusing the advantages of multiple features, and achieves the state-of-the-art performance. However, the deep layers of the DBN-based VAD do not show an apparent superiority to the shallower layers. In this paper, we propose a denoising-deep-neural-network (DDNN) based VAD to address the aforementioned problem. Specifically, we pre-train a deep neural network in a special unsupervised denoising greedy layer-wise mode, and then fine-tune the whole network in a supervised way by the common back-propagation algorithm. In the pre-training phase, we take the noisy speech signals as the visible layer and try to extract a new feature that minimizes the reconstruction cross-entropy loss between the noisy speech signals and its corresponding clean speech signals. Experimental results show that the proposed DDNN-based VAD not only outperforms the DBN-based VAD but also shows an apparent performance improvement of the deep layers over shallower layers.

Machine Learning,Sound
Boosted deep neural networks and multi-resolution cochleagram features for voice activity detection.

Xiao-Lei Zhang,DeLiang Wang

DOI: https://doi.org/10.21437/interspeech.2014-367

2014-01-01

Abstract:Voice activity detection (VAD) is an important frontend of many speech processing systems. In this paper, we describe a new VAD algorithm based on boosted deep neural networks (bDNNs). The proposed algorithm first generates multiple base predictions for a single frame from only one DNN and then aggregates the base predictions for a better prediction of the frame. Moreover, we employ a new acoustic feature, multi-resolution cochleagram (MRCG), that concatenates the cochleagram features at multiple spectrotemporal resolutions and shows superior speech separation results over many acoustic features. Experimental results show that bDNN-based VAD with the MRCG feature outperforms state-of-the-art VADs by a considerable margin.
Audio Sentiment Analysis by Heterogeneous Signal Features Learned from Utterance-Based Parallel Neural Network.

Ziqian Luo,Hua Xu,Feiyang Chen

DOI: https://doi.org/10.29007/7mhj

2019-01-01

Abstract:Audio Sentiment Analysis is a popular research area which extends the text-based sentiment analysis to depend on effectiveness of acoustic features extracted from speech. However, current progress on audio sentiment analysis mainly focuses on extracting homogeneous acoustic features or doesn't fuse heterogeneous features effectively. In this paper, we propose an utterance-based deep neural network model, which has a parallel combination of CNN and LSTM based network, to obtain representative features termed Audio Sentiment Vector (ASV), that can maximally reflect sentiment information in an audio. Specifically, our model is trained by utterance-level labels and ASV can be extracted and fused creatively from two branches. In the CNN model branch, spectrum graphs produced by signals are fed as inputs while in the LSTM model branch, inputs include spectral centroid, MFCC and other recognized traditional acoustic features extracted from dependent utterances in an audio. Besides, BiLSTM with attention mechanism is used for feature fusion. Extensive experiments have been conducted to show our model can recognize audio sentiment precisely, and demonstrate our ASV are better than traditional acoustic features or vectors extracted from other deep learning models. Furthermore, experimental results indicate that the proposed model outperforms state-of-the-art approaches by 9.33% on MOSI.
Boosting Contextual Information for Deep Neural Network Based Voice Activity Detection

Xiao-Lei Zhang,DeLiang Wang

DOI: https://doi.org/10.1109/taslp.2015.2505415

2015-01-01

IEEE/ACM Transactions on Audio Speech and Language Processing

Abstract:Voice activity detection (VAD) is an important topic in audio signal processing. Contextual information is important for improving the performance of VAD at low signal-to-noise ratios. Here we explore contextual information by machine learning methods at three levels. At the top level, we employ an ensemble learning framework, named multi-resolution stacking (MRS), which is a stack of ensemble classifiers. Each classifier in a building block inputs the concatenation of the predictions of its lower building blocks and the expansion of the raw acoustic feature by a given window (called a resolution). At the middle level, we describe a base classifier in MRS, named boosted deep neural network (bDNN). bDNN first generates multiple base predictions from different contexts of a single frame by only one DNN and then aggregates the base predictions for a better prediction of the frame, and it is different from computationally-expensive boosting methods that train ensembles of classifiers for multiple base predictions. At the bottom level, we employ the multi-resolution cochleagram feature, which incorporates the contextual information by concatenating the cochleagram features at multiple spectrotemporal resolutions. Experimental results show that the MRS-based VAD outperforms other VADs by a considerable margin. Moreover, when trained on a large amount of noise types and a wide range of signal-to-noise ratios, the MRS-based VAD demonstrates surprisingly good generalization performance on unseen test scenarios, approaching the performance with noise-dependent training.
Utterance-Based Audio Sentiment Analysis Learned by a Parallel Combination of CNN and LSTM.

Ziqian Luo,Hua Xu,Feiyang Chen

2018-01-01

Abstract:Audio Sentiment Analysis is a popular research area which extends the conventional text-based sentiment analysis to depend on the effectiveness of acoustic features extracted from speech. However, current progress on audio sentiment analysis mainly focuses on extracting homogeneous acoustic features or doesn't fuse heterogeneous features effectively. In this paper, we propose an utterance-based deep neural network model, which has a parallel combination of Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) based network, to obtain representative features termed Audio Sentiment Vector (ASV), that can maximally reflect sentiment information in an audio. Specifically, our model is trained by utterance-level labels and ASV can be extracted and fused creatively from two branches. In the CNN model branch, spectrum graphs produced by signals are fed as inputs while in the LSTM model branch, inputs include spectral features and cepstrum coefficient extracted from dependent utterances in audio. Besides, Bidirectional Long Short-Term Memory (BiLSTM) with attention mechanism is used for feature fusion. Extensive experiments have been conducted to show our model can recognize audio sentiment precisely and quickly, and demonstrate our ASV is better than traditional acoustic features or vectors extracted from other deep learning models. Furthermore, experimental results indicate that the proposed model outperforms the state-of-the-art approach by 9.33\% on Multimodal Opinion-level Sentiment Intensity dataset (MOSI) dataset.
DNN-based Voice Activity Detection for Speaker Recognition

Fanhu Bie,Zhiyong Zhang,Dong Wang,Thomas Fang Zheng

2015-01-01

Abstract:Correspondence: biefh@cslt.riit.tsinghua.edu.cn Center for Speech and Language Technology, Research Institute of Information Technology, Tsinghua University, ROOM 1-303, BLDG FIT, 100084 Beijing, China Department of Computer Science and Technology, Tsinghua University, ROOM 1-303, BLDG FIT, 100084 Beijing, China Full list of author information is available at the end of the article Abstract Voice activity detection (VAD) plays an important role in speaker recognition. This paper proposes to use a novel DNN-based VAD which harnesses the power of deep neural networks (DNN) in learning speech patterns from a large labelled database designed for speech recognition, and thus deliberately optimizes the discrimination between speech and non-speech signals. More interestingly, the output of the DNN offers a noise prior, which may lend itself to a Bayesian treatment for the uncertainty of noise in speaker recognition. The experiments were conducted on the mismatched-microphone condition (C3) of the SRE08 core test. It was found that the DNN-based VAD offered a relative reduction of 22.0% in equal error rate (EER) when compared to a fine-tuned energy-based VAD. When the Bayesian approach was employed, additional gains were obtained, particularly in noise conditions.
Multi-task Joint-Learning for Robust Voice Activity Detection

Yimeng Zhuang,Sibo Tong,Maofan Yin,Yanmin Qian,Kai Yu

DOI: https://doi.org/10.1109/iscslp.2016.7918383

2016-01-01

Abstract:Model based VAD approaches have been widely used and achieved success in practice. These approaches usually cast VAD as a frame-level classification problem and employ statistical classifiers, such as Gaussian Mixture Model (GMM) or Deep Neural Network (DNN) to assign a speech/silence label for each frame. Due to the frame independent assumption classification, the VAD results tend to be fragile. To address this problem, in this paper, a new structured multi-frame prediction DNN approach is proposed to improve the segment-level VAD performance. During DNN training, VAD labels of multiple consecutive frames are concatenated together as targets and jointly trained with a speech enhancement task to achieve robustness under noisy conditions. During testing, the VAD label for each frame is obtained by merging the prediction results from neighbouring frames. Experiments on an Aurora 4 dataset showed that, conventional DNN based VAD has poor and unstable prediction performance while the proposed multitask trained VAD is much more robust.
Efficient voice activity detection algorithm based on sub-band temporal envelope and sub-band long-term signal variability

Bin Liu,Jianhua Tao,Fuyuan Mo,Ya Li,Zhengqi Wen,Shanfeng Liu

DOI: https://doi.org/10.1109/ISCSLP.2014.6936602

2014-01-01

Abstract:Voice activity detection (VAD) is widely used for various speech-based systems which is an important pre-processing step. This paper proposes a robust voice activity detection algorithm. In the proposed algorithm, the sub-band temporal envelope and the sub-band long-term signal variability are considered to distinguish the speech from all kinds of non-speech which include stationary noise and non-stationary noise. The two features are combined to make a robust VAD decision according to the fusion decision. The proposed algorithm also is an unsupervised low-complexity algorithm and can operate without pre-train models. The experiments results show that the proposed algorithm is prior to the different baseline algorithms and can handle a variety of noise environments over a wide range of signal-to-noise ratios. The proposed algorithm could apply to speech-based systems.
A Universal VAD Based on Jointly Trained Deep Neural Networks.

Qing Wang,Jun Du,Xiao Bao,Zi-Rui Wang,Li-Rong Dai,Chin-Hui Lee

DOI: https://doi.org/10.21437/interspeech.2015-442

2015-01-01

Abstract:In this paper, we propose a joint training approach to voice activity detection (VAD) to address the issue of performance degradation due to unseen noise conditions. Two key techniques are integrated into this deep neural network (DNN) based VAD framework. First, a regression DNN is trained to map the noisy to clean speech features similar to DNN-based speech enhancement. Second, the VAD part to discriminate speech against noise backgrounds is also a DNN trained with a large amount of diversified noisy data synthesized by a wide range of additive noise types. By stacking the classification DNN on top of the enhancement DNN, this integrated DNN can be jointly trained to perform VAD. The feature mapping DNN serves as a noise normalization module aiming at explicitly generating the "clean" features which are easier to be correctly recognized by the following classification DNN. Our experiment results demonstrate the proposed noise-universal DNN-based VAD algorithm achieves a good generalization capacity to unseen noises, and the jointly trained DNNs consistently and significantly outperform the conventional classification-based DNN for all the noise types and signal-to-noise levels tested.
Transfer Learning for Voice Activity Detection: A Denoising Deep Neural Network Perspective

Xiao-Lei Zhang,Ji Wu

DOI: https://doi.org/10.48550/arXiv.1303.2104

IF: 5.414

2013-03-08

Machine Learning

Abstract:Mismatching problem between the source and target noisy corpora severely hinder the practical use of the machine-learning-based voice activity detection (VAD). In this paper, we try to address this problem in the transfer learning prospective. Transfer learning tries to find a common learning machine or a common feature subspace that is shared by both the source corpus and the target corpus. The denoising deep neural network is used as the learning machine. Three transfer techniques, which aim to learn common feature representations, are used for analysis. Experimental results demonstrate the effectiveness of the transfer learning schemes on the mismatch problem.
Improved Voice Activity Detection Based on Long-term Spectral Divergence and Pitch Ratio Features

MENG Yi-ming,OU Zhi-jian

DOI: https://doi.org/10.3969/j.issn.1001-893x.2013.08.013

2013-01-01

Abstract:Voice Activity Detection(VAD) is the front-end of speech processing and the VAD algorithm which uses long-term spectral divergence(LTSD) feature can′t discriminate abrupt noise from speech.The speech signal with abrupt noise will adversely affect the speech processing system.This paper proposes a VAD algorithm which combines LTSD feature and pitch ratio feature.The advantage of the algorithm is that by introducing pitch ratio feature,it can effectively reduce the false alarms of taking abrupt noise as speech.Experimental results show that the algorithm achieves good performance for VAD under various signal-to-noise ratios.
Unsupervised domain adaptation for deep neural network based voice activity detection

Xiao-Lei Zhang

DOI: https://doi.org/10.1109/ICASSP.2014.6854930

2014-01-01

ICASSP

Abstract:The mismatching problem between the training and test speech corpora hinders the practical use of the machine-learning-based voice activity detection (VAD). In this paper, we try to address this problem by the unsupervised domain adaptation techniques, which try to find a shared feature subspace between the mismatching corpora. The denoising deep neural network is used as the learning machine. Three domain adaptation techniques are used for analysis. Experimental results show that the unsupervised domain adaptation technique is promising to the mismatching problem of VAD.
A Real-Time Voice Activity Detection Based On Lightweight Neural

Jidong Jia,Pei Zhao,Di Wang

2024-05-27

Abstract:Voice activity detection (VAD) is the task of detecting speech in an audio stream, which is challenging due to numerous unseen noises and low signal-to-noise ratios in real environments. Recently, neural network-based VADs have alleviated the degradation of performance to some extent. However, the majority of existing studies have employed excessively large models and incorporated future context, while neglecting to evaluate the operational efficiency and latency of the models. In this paper, we propose a lightweight and real-time neural network called MagicNet, which utilizes casual and depth separable 1-D convolutions and GRU. Without relying on future features as input, our proposed model is compared with two state-of-the-art algorithms on synthesized in-domain and out-domain test datasets. The evaluation results demonstrate that MagicNet can achieve improved performance and robustness with fewer parameter costs.

Sound,Artificial Intelligence,Audio and Speech Processing
A Robust, Real-Time Voice Activity Detection Algorithm for Embedded Mobile Devices.

Bian Wu,Xiaolin Ren,Chongqing Liu,Yaxin Zhang

DOI: https://doi.org/10.1007/s10772-005-2165-7

1997-01-01

Journal of Sol-Gel Science and Technology

Abstract:When an Automatic Speech Recognition (ASR) system is applied in noisy environments, Voice Activity Detection (VAD) is crucial to the performance of the overall system. The employment of the VAD for ASR on embedded mobile systems will minimize physical distractions and make the system convenient to use. Conventional VAD algorithm is of high complexity, which makes it unsuitable for embedded mobile devices; or of low robustness, which holds back its application in mobile noisy environments. In this paper, we propose a robust VAD algorithm specifically designed for ASR on embedded mobile devices. The architecture of the proposed algorithm is based on a two-level decision making strategy, where there is an interaction between a lower features-based level and subsequent decision logic based on a finite-state machine. Many discriminating features are employed in the lower level to improve the robustness of the VAD. The two-level decision strategy allows different features to be used in different states and reduces the cost of the algorithm, which makes the proposed algorithm suitable for embedded mobile devices. The evaluation experiments show the proposed VAD algorithm is robust and contribute to the overall performance gain of the ASR system in various acoustic environments.
Voice Activity Detection Based on Time-Delay Neural Networks

Ye Bai,Jiangyan Yi,Jianhua Tao,Zhengqi Wen,Bin Liu

DOI: https://doi.org/10.1109/apsipaasc47483.2019.9023262

2019-01-01

Abstract:Voice activity detection (VAD) is an important preprocessing part of many speech applications. Context information is important for VAD. Time-delay neural networks (TDNNs) capture long context information with a few parameters. This paper investigates a TDNN based VAD framework. A simple chunk based decision method is proposed to smooth raw posteriors and decide border points of utterances. To evaluate decision performance, a metric intersection-over-union (IoU) is introduced from image object detection. The experiment results are evaluated on Wall Street Journal (WSJ0) corpus. Frame classification performance is measured by area under the curve (AUC) and equal error rate (EER). Compared with long short-term memory baseline, the TDNN based system achieves a 41.26% EER relative reduction on average in matched noise condition, and relative improvement of average AUC is 3.82%. Proposed decision method achieves an 18.74% IoU relative improvement on average compared with moving average method on average.
Audio-visual voice activity detection using diffusion maps

David Dov,Ronen Talmon,Israel Cohen

DOI: https://doi.org/10.1109/TASLP.2015.2405481

2015-04-01

Abstract:The performance of traditional voice activity detectors significantly deteriorates in the presence of highly nonstationary noise and transient interferences. One solution is to incorporate a video signal which is invariant to the acoustic environment. Although several voice activity detectors based on the video signal were recently presented, merely few detectors which are based on both the audio and the video signals exist in the literature to date. In this paper, we present an audio-visual voice activity detector and show that the incorporation of both audio and video signals is highly beneficial for voice activity detection. The algorithm is based on a supervised learning procedure, and a labeled training data set is considered. The algorithm comprises a feature extraction procedure, where the features are designed to separate speech from nonspeech frames. Diffusion maps is applied separately and similarly to the features of each modality and builds a low dimensional representation. Using the new representation, we propose a measure for voice activity which is based on a supervised learning procedure and the variability between adjacent frames in time. The measures of the two modalities are merged to provide voice activity detection based on both the audio and the video signals. Experimental results demonstrate the improved performance of the proposed algorithm compared to state-of-the-art detectors.
Audio-Visual Information Fusion Using Cross-Modal Teacher-Student Learning for Voice Activity Detection in Realistic Environments

Hengshun Zhou,Jun Du,Hang Chen,Zijun Jing,Shifu Xiong,Chin-Hui Lee

DOI: https://doi.org/10.21437/interspeech.2021-592

2021-01-01

Abstract:We propose an information fusion approach to audio-visual voice activity detection (AV-VAD) based on cross-modal teacher-student learning leveraging on factorized bilinear pooling (FBP) and Kullback-Leibler (KL) regularization. First, we design an audio-visual network by using FBP fusion to fully utilize the interaction between audio and video modalities. Next, to transfer the rich information in audio-based VAD (A-VAD) model trained with a massive audio-only dataset to AV-VAD model built with relatively limited multi-modal data, a cross-modal teacher-student learning framework is then proposed based on cross entropy with regulated KL-divergence. Finally, evaluated on an in-house dataset recorded in realistic conditions using standard VAD metrics, the proposed approach yields consistent and significant improvements over other state-of-the-art techniques. Moreover, by applying our AV-VAD technique to an audio-visual Chinese speech recognition task, the character error rate is reduced by 24.15% and 8.66% from AVAD and the baseline AV-VAD systems, respectively.

Deep Belief Networks Based Voice Activity Detection

Deep Belief Networks Based Voice Activity Detection

A Novel and Efficient Voice Activity Detector Using Shape Features of Speech Wave.

Applying Support Vector Machines to Voice Activity Detection

Denoising Deep Neural Networks Based Voice Activity Detection

Boosted deep neural networks and multi-resolution cochleagram features for voice activity detection.

Audio Sentiment Analysis by Heterogeneous Signal Features Learned from Utterance-Based Parallel Neural Network.

Boosting Contextual Information for Deep Neural Network Based Voice Activity Detection

Utterance-Based Audio Sentiment Analysis Learned by a Parallel Combination of CNN and LSTM.

DNN-based Voice Activity Detection for Speaker Recognition

Multi-task Joint-Learning for Robust Voice Activity Detection

Efficient voice activity detection algorithm based on sub-band temporal envelope and sub-band long-term signal variability

A Universal VAD Based on Jointly Trained Deep Neural Networks.

Transfer Learning for Voice Activity Detection: A Denoising Deep Neural Network Perspective

Improved Voice Activity Detection Based on Long-term Spectral Divergence and Pitch Ratio Features

Unsupervised domain adaptation for deep neural network based voice activity detection

A Real-Time Voice Activity Detection Based On Lightweight Neural

A Robust, Real-Time Voice Activity Detection Algorithm for Embedded Mobile Devices.

Voice Activity Detection Based on Time-Delay Neural Networks

Audio-visual voice activity detection using diffusion maps

Audio-Visual Information Fusion Using Cross-Modal Teacher-Student Learning for Voice Activity Detection in Realistic Environments