Abstract:Singing melody extraction from polyphonic musical audio is one of the most challenging tasks in music information retrieval (MIR). Recently, data-driven methods based on convolutional neural networks (CNNs) have achieved great success for this task. In the literature, harmonic relationship has been proven crucial for this task. However, few existing CNN-based singing melody extraction methods consider the harmonic relationship in the training stage. The state-of-the-art CNN based methods are not capable of capturing such long-dependency harmonic relationship due to limited receptive field and unacceptable computation cost. In this paper, we introduce a neural harmonic-aware network with gated attentive fusion (NHAN-GAF) for singing melody extraction. Specifically, in the 2-D spectrograms modeling branch, we propose to employ multiple parallel 1-D CNN kernels to capture the harmonic relations between 1–2 octaves along the frequency axis in the spectrogram. Considering the advantage of jointly using Time–Frequency (T-F) domain and time domain information, we use two-branch neural nets to learn discriminative representation for this task. A novel gated attentive fusion (GAF) network is suggested to encode potential correlations between the two branches and fuse the descriptors learned from raw waveform and T-F spectrograms. Moreover, the idea of GAF can be exploited to the multimedia applications with multimodal analysis. With the two proposed components, our proposed model is capable of learning the harmonic relationship in the spectrogram and better capturing the contextual but discriminative features for singing melody extraction. We use part of the vocal tracks of the RWC dataset and MIR-1 K dataset to train the model and evaluate the performance of the proposed model on the ADC 2004, MIREX 05 and MedleyDB datasets. The experimental results show that the proposed method outperforms the state-of-the-art ones.

HarmoF0: Logarithmic Scale Dilated Convolution for Pitch Estimation

Vocal Melody Extraction Via Dnn-Based Pitch Estimation And Salience-Based Pitch Refinement

MF-PAM: Accurate Pitch Estimation through Periodicity Analysis and Multi-level Feature Fusion

Polyphonic pitch detection with convolutional recurrent neural networks

Towards Improving Harmonic Sensitivity and Prediction Stability for Singing Melody Extraction

Robust Multipitch Estimation Of Piano Sounds Using Deep Spiking Neural Networks

Toward Fully Self-Supervised Multi-Pitch Estimation

A Robust and Low Computational Cost Pitch Estimation Method

HPPNet: Modeling the Harmonic Structure and Pitch Invariance in Piano Transcription

A neural harmonic-aware network with gated attentive fusion for singing melody extraction

Pitch-scaled Analysis Based Residual Reconstruction for Speech Analysis and Synthesis

Unsupervised Harmonic Parameter Estimation Using Differentiable DSP and Spectral Optimal Transport

Speech Enhancement Based On Analysis Synthesis Framework With Improved Pitch Estimation And Spectral Envelope Enhancement

SNR-Progressive Model with Harmonic Compensation for Low-SNR Speech Enhancement

Multiple Fundamental Frequency Estimation by Modeling Spectral Peaks and Non-Peak Regions

TONet: Tone-Octave Network for Singing Melody Extraction from Polyphonic Music

Human Voice Pitch Estimation: A Convolutional Network with Auto-Labeled and Synthetic Data

Inferring Pitch from Coarse Spectral Features

Pushing the Limit of Sound Event Detection with Multi-Dilated Frequency Dynamic Convolution

Neural Harmonium: An Interpretable Deep Structure for Nonlinear Dynamic System Identification with Application to Audio Processing

An hrnet-blstm model with two-stage training for singing melody extraction