Abstract:Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as 'ing' or whole words. Recent literature has repeatedly shown the limitations of such a tokenization strategy, particularly for documents not written in English and for representing numbers. On the other extreme, byte/character-level language models are much less restricted but suffer from increased sequence description lengths and a subsequent quadratic expansion in self-attention computation. Recent attempts to compress and limit these context lengths with fixed size convolutions is helpful but completely ignores the word boundary. This paper considers an alternative 'learn your tokens' scheme which utilizes the word boundary to pool bytes/characters into word representations, which are fed to the primary language model, before again decoding individual characters/bytes per word in parallel. We find that our moderately expressive and moderately fast end-to-end tokenizer outperform by over 300% both subwords and byte/character models over the intrinsic language modeling metric of next-word prediction across datasets. It particularly outshines on rare words, outperforming by a factor of 30! We extensively study the language modeling setup for all three categories of tokenizers and theoretically analyze how our end-to-end models can also be a strong trade-off in efficiency and robustness.

SHUOWEN-JIEZI: Linguistically Informed Tokenizers For Chinese Language Model Pretraining

Sub-Character Tokenization for Chinese Pretrained Language Models

ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information

Shuo Wen Jie Zi: Rethinking Dictionaries and Glyphs for Chinese Language Pre-training

Character, Word, or Both? Revisiting the Segmentation Granularity for Chinese Pre-trained Language Models

Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models

Exploiting Word Semantics to Enrich Character Representations of Chinese Pre-trained Models

JiuZhang: A Chinese Pre-trained Language Model for Mathematical Problem Understanding

Pre-Training with Whole Word Masking for Chinese BERT

"Is Whole Word Masking Always Better for Chinese BERT?": Probing on Chinese Grammatical Error Correction

Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning

Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languages

Unsupervised Chinese Word Segmentation with BERT Oriented Probing and Transformation

Learn Your Tokens: Word-Pooled Tokenization for Language Modeling

JiuZhang 2.0: A Unified Chinese Pre-trained Language Model for Multi-task Mathematical Problem Solving

An Interpretability Evaluation Benchmark for Pre-trained Language Models

StyleBERT: Chinese pretraining by font style information

Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models

Pretraining without wordpieces: learning over a vocabulary of millions of words

Character-Level Syntax Infusion in Pre-Trained Models for Chinese Semantic Role Labeling.

CharBERT: Character-aware Pre-trained Language Model