Abstract:Despite the recent increase in research on artificial intelligence for music, prominent correlations between key components of lyrics and rhythm such as keywords, stressed syllables, and strong beats are not frequently studied. This is likely due to challenges such as audio misalignment, inaccuracies in syllabic identification, and most importantly, the need for cross-disciplinary knowledge. To address this lack of research, we propose a novel multimodal lyrics-rhythm matching approach in this paper that specifically matches key components of lyrics and music with each other without any language limitations. We use audio instead of sheet music with readily available metadata, which creates more challenges yet increases the application flexibility of our method. Furthermore, our approach creatively generates several patterns involving various multimodalities, including music strong beats, lyrical syllables, auditory changes in a singer's pronunciation, and especially lyrical keywords, which are utilized for matching key lyrical elements with key rhythmic elements. This advantageous approach not only provides a unique way to study auditory lyrics-rhythm correlations including efficient rhythm-based audio alignment algorithms, but also bridges computational linguistics with music as well as music cognition. Our experimental results reveal an 0.81 probability of matching on average, and around 30% of the songs have a probability of 0.9 or higher of keywords landing on strong beats, including 12% of the songs with a perfect landing. Also, the similarity metrics are used to evaluate the correlation between lyrics and rhythm. It shows that nearly 50% of the songs have 0.70 similarity or higher. In conclusion, our approach contributes significantly to the lyrics-rhythm relationship by computationally unveiling insightful correlations.

Similarity of structures in popular music

The Interconnections of Music Structure, Harmony, Melody, Rhythm, and Predictivity

In-depth analysis of music structure as a text network

Cross-Similarity Measurement of Music Sections: A Framework for Large-scale Cover Song Identification

Grouping compositions based on similarity of music themes

Barwise Music Structure Analysis with the Correlation Block-Matching Segmentation Algorithm

Analysis and Visualization of Musical Structure using Networks

Do we need more complex representations for structure? A comparison of note duration representation for Music Transformers

Symbolic Music Structure Analysis with Graph Representations and Changepoint Detection Methods

On the Complex Network Structure of Musical Pieces: Analysis of Some Use Cases from Different Music Genres

Automatic Analysis and Influence of Hierarchical Structure on Melody, Rhythm and Harmony in Popular Music

Convolutive Block-Matching Segmentation Algorithm with Application to Music Structure Analysis

What is missing in deep music generation? A study of repetition and structure in popular music

Songs tell a story: The Arc of narrative for music

Headphones on the wire

The Skipping Behavior of Users of Music Streaming Services and its Relation to Musical Structure

Structural characterization of musical harmonies

Quantifying Musical Style: Ranking Symbolic Music based on Similarity to a Style

Music-Circles: Can Music Be Represented With Numbers?

Multimodal Lyrics-Rhythm Matching

Court decisions on music plagiarism and the predictive value of similarity algorithms