Abstract:When used with deep learning, the symbolic music modality is often coupled with language model architectures. To do so, the music needs to be tokenized, i.e. converted into a sequence of discrete tokens. This can be achieved by different approaches, as music can be composed of simultaneous tracks, of simultaneous notes with several attributes. Until now, the proposed tokenizations rely on small vocabularies of tokens describing the note attributes and time events, resulting in fairly long token sequences, and a sub-optimal use of the embedding space of language models. Recent research has put efforts on reducing the overall sequence length by merging embeddings or combining tokens. In this paper, we show that Byte Pair Encoding, a compression technique widely used for natural language, significantly decreases the sequence length while increasing the vocabulary size. By doing so, we leverage the embedding capabilities of such models with more expressive tokens, resulting in both better results and faster inference in generation and classification tasks. The source code is shared on Github, along with a companion website. Finally, BPE is directly implemented in MidiTok, allowing the reader to easily benefit from this method.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to optimize the representation of symbolic music by using Byte Pair Encoding (BPE) technology, thereby improving the quality and efficiency of generation and classification tasks. Specifically, the author points out some limitations of existing methods in dealing with symbolic music: 1. **Long - sequence problem**: Existing tokenization methods usually rely on tokens in a small vocabulary to describe note attributes and time events, which results in longer token sequences. Long sequences not only increase computational complexity but also limit the effective use of the language model's embedding space. 2. **Insufficient information expression**: Current tokenization methods are only based on simple note attributes and time events. These tokens themselves carry limited information and cannot fully utilize the powerful embedding capabilities of the language model. 3. **Inefficient use of embedding space**: The embedding dimensions used by most existing methods are much higher than the vocabulary size, resulting in under - utilization of the embedding space and affecting model performance. To solve these problems, the author introduces BPE technology, whose main goals are: - **Reduce sequence length**: Significantly reduce the length of the token sequence by merging frequently occurring byte pairs, thereby improving the model's inference speed and efficiency. - **Increase vocabulary size**: Expand the vocabulary to include more expressive tokens, so that each token can carry more information, thereby making better use of the language model's embedding capabilities. - **Improve the performance of generation and classification tasks**: Improve the quality of generated music and the accuracy of classification tasks through a more effective tokenization method. ### Main contributions of the paper 1. **Applying BPE to symbolic music**: For the first time, BPE is applied to the field of symbolic music, demonstrating its effectiveness in reducing sequence length and improving model performance. 2. **Experimental verification**: Through experiments on multiple public datasets, it is proved that BPE is superior to other sequence compression techniques in generation and classification tasks. 3. **Geometric analysis**: The spatial distribution characteristics of the learned embedding vectors are studied, further verifying the advantages of BPE. 4. **Open - source implementation**: The source code and supporting website are provided to facilitate other researchers to reproduce and apply this method. In summary, this paper aims to improve the tokenization method of symbolic music by introducing BPE technology, thereby enhancing the performance of deep - learning models in music generation and classification tasks.

Byte Pair Encoding for Symbolic Music

Analyzing Byte-Pair Encoding on Monophonic and Polyphonic Symbolic Music: A Focus on Musical Phrase Segmentation

From Words to Music: A Study of Subword Tokenization Techniques in Symbolic Music Generation

Impact of time and note duration tokenizations on deep learning symbolic music modeling

PiRhDy: Learning Pitch-, Rhythm-, and Dynamics-aware Embeddings for Symbolic Music.

Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generation

A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modeling

PerTok: Expressive Encoding and Modeling of Symbolic Musical Ideas and Variations

N-Gram Unsupervised Compoundation and Feature Injection for Better Symbolic Music Understanding

Practical and Reproducible Symbolic Music Generation by Large Language Models with Structural Embeddings

From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities

A Formal Perspective on Byte-Pair Encoding

Nested Music Transformer: Sequentially Decoding Compound Tokens in Symbolic Music and Audio Generation

miditok: A Python package for MIDI file tokenization

Theoretical Analysis of Byte-Pair Encoding

Coordinate Embedding Transformer Model for Optical Music Recognition on Monophonic Scores

End-to-end Piano Performance-MIDI to Score Conversion with Transformers

Systematic Analysis of Music Representations from BERT

MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training

ByteComposer: a Human-like Melody Composition Method based on Language Model Agent

Improved symbolic drum style classification with grammar-based hierarchical representations