Abstract:The question of whether people's experience in the world shapes conceptual representation and lexical semantics is longstanding. Word-association, feature-listing and similarity rating tasks aim to address this question but require a subjective interpretation of the latent dimensions identified. In this study, we introduce a supervised representational-alignment method that (i) determines whether two groups of individuals share the same basis of a certain category, and (ii) explains in what respects they differ. In applying this method, we show that congenital blindness induces conceptual reorganization in both a-modal and sensory-related verbal domains, and we identify the associated semantic shifts. We first apply supervised feature-pruning to a language model (GloVe) to optimize prediction accuracy of human similarity judgments from word embeddings. Pruning identifies one subset of retained GloVe features that optimizes prediction of judgments made by sighted individuals and another subset that optimizes judgments made by blind. A linear probing analysis then interprets the latent semantics of these feature-subsets by learning a mapping from the retained GloVe features to 65 interpretable semantic dimensions. We applied this approach to seven semantic domains, including verbs related to motion, sight, touch, and amodal verbs related to knowledge acquisition. We find that blind individuals more strongly associate social and cognitive meanings to verbs related to motion or those communicating non-speech vocal utterances (e.g., whimper, moan). Conversely, for amodal verbs, they demonstrate much sparser information. Finally, for some verbs, representations of blind and sighted are highly similar. The study presents a formal approach for studying interindividual differences in word meaning, and the first demonstration of how blindness impacts conceptual representation of everyday verbs.

A Computational Model of Concept Generalization in Cross-Modal Reference

On the Complexity of Bayesian Generalization

Modeling Conceptual Understanding in Image Reference Games

Exploring the Effect of Primitives for Compositional Generalization in Vision-and-Language

Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization

Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension

Identifying and interpreting non-aligned human conceptual representations using language modeling

Cognitive Principles in Robust Multimodal Interpretation

Extending Machine Language Models toward Human-Level Language Understanding

Modelling Multimodal Integration in Human Concept Processing with Vision-and-Language Models

Understanding Visual Concepts Across Models

The Interaction of Memory and Attention in Novel Word Generalization: A Computational Investigation

Conceptual and Unbiased Reasoning in Language Models

A Study of Compositional Generalization in Neural Models

A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces

A Universal Model for Cross Modality Mapping by Relational Reasoning

Obtaining a Figurative Interpretation of a Word: Support for Underspecification

The Role of Linguistic Priors in Measuring Compositional Generalization of Vision-Language Models

Understanding Inter-Concept Relationships in Concept-Based Models

From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency