Abstract:An automatic synchronization system of the popular song and its lyrics is presented in the paper. The system includes two main components: a) automatically detecting vocal/non-vocal in the audio signal and b) automatically aligning the acoustic signal of the song with its lyric using speech recognition techniques and positioning the boundaries of the lyrics in its acoustic realization at the multiple levels simultaneously (e.g. the word / syllable level and phrase level). The GMM models and a set of HMM-based acoustic model units are carefully designed and trained for the detection and alignment. To eliminate the severe mismatch due to the diversity of musical signal and sparse training data available, the unsupervised adaptation technique such as maximum likelihood linear regression (MLLR) is exploited for tailoring the models to the real environment, which improves robustness of the synchronization system. To further reduce the effect of the missed non-vocal music on alignment, a novel grammar net is build to direct the alignment. As we know, this is the first automatic synchronization system only based on the low-level acoustic feature such as MFCC. We evaluate the system on a Chinese song dataset collecting from 3 popular singers. We obtain 76.1% for the boundary accuracy at the syllable level (BAS) and 81.5% for the boundary accuracy at the phrase level (BAP) using fully automatic vocal/non-vocal detection and alignment. The synchronization system has many applications such as multi-modality (audio and textual) content-based popular song browsing and retrieval. Through the study, we would like to open up the discussion of some challenging problems when developing a robust synchronization system for largescale database.

A Robust Audio Similarity Estimation Method for Audio Alignment in Mobile Karaoke Apps

A Cross-Modal Approach for Karaoke Artifacts Correction

Audio to Score Alignment Based on Chroma Features and Dynamic Time Warping Algorithm

Efficient Karaoke Song Recommendation Via Multiple Kernel Learning Approximation

The design and implementation of a smartphone-based acoustic array system for DOA estimation

A Real-Time Lyrics Alignment System Using Chroma And Phonetic Features For Classical Vocal Performance

On-Line Audio-to-Lyrics Alignment Based on a Reference Performance

Popular song and lyrics synchronization and its application to music information retrieval

A Channel and Carrier Frequency Offset Joint Estimation Algorithm for TD-SCDMA

Rhythmic Foley: A Framework For Seamless Audio-Visual Alignment In Video-to-Audio Synthesis

AlignNet: A Unifying Approach to Audio-Visual Alignment

Multimodal content-based structure analysis of karaoke music.

Just Label the Repeats for In-The-Wild Audio-to-Score Alignment

DALI: a large Dataset of synchronized Audio, LyrIcs and notes, automatically created using teacher-student machine learning paradigm

Mutual Alignment between Audiovisual Features for End-to-End Audiovisual Speech Recognition

AutoMatch: A Large-scale Audio Beat Matching Benchmark for Boosting Deep Learning Assistant Video Editing

An Effective Vocal/Non-vocal Segmentation Approach for Embedded Music Retrieve System on Mobile Phone

EventDTW: An Improved Dynamic Time Warping Algorithm for Aligning Biomedical Signals of Nonuniform Sampling Frequencies

HCLAS-X: Hierarchical and Cascaded Lyrics Alignment System Using Multimodal Cross-Correlation

Community-based matrix factorization for scalable music recommendation on smartphones

Aligning Audio-Visual Joint Representations with an Agentic Workflow