Abstract:Generative recommendation systems, driven by large language models (LLMs), present an innovative approach to predicting user preferences by modeling items as token sequences and generating recommendations in a generative manner. A critical challenge in this approach is the effective tokenization of items, ensuring that they are represented in a form compatible with LLMs. Current item tokenization methods include using text descriptions, numerical strings, or sequences of discrete tokens. While text-based representations integrate seamlessly with LLM tokenization, they are often too lengthy, leading to inefficiencies and complicating accurate generation. Numerical strings, while concise, lack semantic depth and fail to capture meaningful item relationships. Tokenizing items as sequences of newly defined tokens has gained traction, but it often requires external models or algorithms for token assignment. These external processes may not align with the LLM's internal pretrained tokenization schema, leading to inconsistencies and reduced model performance. To address these limitations, we propose a self-improving item tokenization method that allows the LLM to refine its own item tokenizations during training process. Our approach starts with item tokenizations generated by any external model and periodically adjusts these tokenizations based on the LLM's learned patterns. Such alignment process ensures consistency between the tokenization and the LLM's internal understanding of the items, leading to more accurate recommendations. Furthermore, our method is simple to implement and can be integrated as a plug-and-play enhancement into existing generative recommendation systems. Experimental results on multiple datasets and using various initial tokenization strategies demonstrate the effectiveness of our method, with an average improvement of 8\% in recommendation performance.

Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond

A Vocabulary-Free Multilingual Neural Tokenizer for End-to-End Task Learning

Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models

Retrofitting (Large) Language Models with Dynamic Tokenization

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization

Enhancing Item Tokenization for Generative Recommendation through Self-Improvement

Token Imbalance Adaptation for Radiology Report Generation

Tokenization Falling Short: On Subword Robustness in Large Language Models

Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning

Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles

Learn Your Tokens: Word-Pooled Tokenization for Language Modeling

Token-level Direct Preference Optimization

Infusing clinical knowledge into tokenisers for language models

One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models

Downstream Task-Oriented Neural Tokenizer Optimization with Vocabulary Restriction as Post Processing

Comparing Variation in Tokenizer Outputs Using a Series of Problematic and Challenging Biomedical Sentences

Probabilistic Linguistic Knowledge and Token-level Text Augmentation

Getting the most out of your tokenizer for pre-training and domain adaptation

Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning

Efficient Sparse Attention needs Adaptive Token Release

Tokenization Is More Than Compression