Abstract:Pretrained language models are long known to be subpar in capturing sentence and document-level semantics. Though heavily investigated, transferring perturbation-based methods from unsupervised visual representation learning to NLP remains an unsolved problem. This is largely due to the discreteness of subword units brought by tokenization of language models, limiting small perturbations of inputs to form semantics-preserved positive pairs. In this work, we conceptualize the learning of sentence-level textual semantics as a visual representation learning process. Drawing from cognitive and linguistic sciences, we introduce an unsupervised visual sentence representation learning framework, employing visually-grounded text perturbation methods like typos and word order shuffling, resonating with human cognitive patterns, and enabling perturbation to texts to be perceived as continuous. Our approach is further bolstered by large-scale unsupervised topical alignment training and natural language inference supervision, achieving comparable performance in semantic textual similarity (STS) to existing state-of-the-art NLP methods. Additionally, we unveil our method's inherent zero-shot cross-lingual transferability and a unique leapfrogging pattern across languages during iterative training. To our knowledge, this is the first representation learning method devoid of traditional language models for understanding sentence and document semantics, marking a stride closer to human-like textual comprehension. Our code is available at https://github.com/gowitheflow-1998/Pixel-Linguist

Evaluating Pixel Language Models on Non-Standardized Languages

Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models

Language Modelling with Pixels

Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languages

To token or not to token: A Comparative Study of Text Representations for Cross-Lingual Transfer

Text Rendering Strategies for Pixel Language Models

A Benchmark for Evaluating Machine Translation Metrics on Dialects Without Standard Orthography

Quantifying the Dialect Gap and its Correlates Across Languages

Pixel Aligned Language Models

Can Multilingual Language Models Transfer to an Unseen Dialect? A Case Study on North African Arabizi

Investigating cross-lingual training for offensive language detection

Evaluating and explaining training strategies for zero-shot cross-lingual news sentiment analysis

CLIPPO: Image-and-Language Understanding from Pixels Only

Autoregressive Pre-Training on Pixels and Texts

Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models

Dialetto, ma Quanto Dialetto? Transcribing and Evaluating Dialects on a Continuum

Low-resource Bilingual Dialect Lexicon Induction with Large Language Models

Small Language Models Like Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamas

Pixel Sentence Representation Learning

Language-Augmented Pixel Embedding for Generalized Zero-Shot Learning