Abstract:It is widely believed that orthographic processing implies an approximate, flexible coding of letter position, as shown by relative-position and transposition priming effects in visual word recognition. These findings have inspired alternative proposals about the representation of letter position, ranging from noisy coding across the ordinal positions to relative position coding based on open bigrams. This debate can be cast within the broader problem of learning location-invariant representations of written words, that is, a coding scheme abstracting the identity and position of letters (and combinations of letters) from their eye-centered (i.e., retinal) locations. We asked whether location-invariance would emerge from deep unsupervised learning on letter strings and what type of intermediate coding would emerge in the resulting hierarchical generative model. We trained a deep network with three hidden layers on an artificial dataset of letter strings presented at five possible retinal locations. Though word-level information (i.e., word identity) was never provided to the network during training, linear decoding from the activity of the deepest hidden layer yielded near-perfect accuracy in location-invariant word recognition. Conversely, decoding from lower layers yielded a large number of transposition errors. Analyses of emergent internal representations showed that word selectivity and location invariance increased as a function of layer depth. Word-tuning and location-invariance were found at the level of single neurons, but there was no evidence for bigram coding. Finally, the distributed internal representation of words at the deepest layer showed higher similarity to the representation elicited by the two exterior letters than by other combinations of two contiguous letters, in agreement with the hypothesis that word edges have special status. These results reveal that the efficient coding of written words-which was the model's learning objective-is largely based on letter-level information.

Referential Uncertainty and Word Learning in High-dimensional, Continuous Meaning Spaces

Word learning under infinite uncertainty

Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training

The Subspace Voyager: Exploring High-Dimensional Data along a Continuum of Salient 3D Subspaces

Local Topology Measures of Contextual Language Model Latent Spaces With Applications to Dialogue Term Extraction

High-dimensional distributed semantic spaces for utterances

Scale-dependent Relationships in Natural Language

Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledge

Towards Resolving Word Ambiguity with Word Embeddings

Exploring the Meaningfulness of Nearest Neighbor Search in High-Dimensional Space

Mapping semantic space: Exploring the higher-order structure of word meaning

Identify Shifts of Word Semantics Through Bayesian Surprise

Emerging Language Spaces Learned From Massively Multilingual Corpora

Deep generative learning of location-invariant visual word recognition

Materializing and Querying Learned Knowledge

Surveying the space of descriptions of a composite system with machine learning

Deep daxes: Mutual exclusivity arises through both learning biases and pragmatic strategies in neural networks

To Word Senses and Beyond: Inducing Concepts with Contextualized Language Models

The Geometry and Dynamics of Meaning

Local similarity and global variability characterize the semantic space of human languages

Low-dimensional Semantic Space: from Text to Word Embedding