Abstract:Background: Language and the ability to communicate effectively are key factors in mental health and well-being. Despite this critical importance, research on language is limited by the lack of a scalable phenotyping toolkit. Methods: Here, we describe and showcase Lingo - a flexible online battery of language and nonverbal reasoning skills based on seven widely used tasks (COWAT, picture narration, vocal rhythm entrainment, rapid automatized naming, following directions, sentence repetition, and nonverbal reasoning). The current version of Lingo takes approximately 30 minutes to complete, is entirely open source, and allows for a wide variety of performance metrics to be extracted. We asked > 1,300 individuals from multiple samples to complete Lingo, then investigated the validity and utility of the resulting data. Results: We conducted an exploratory factor analysis across 14 features derived from the seven assessments, identifying five factors. Four of the five factors showed acceptable test-retest reliability (Pearson's R > 0.7). Factor 2 showed the highest reliability (Pearson's R = 0.95) and loaded primarily on sentence repetition task performance. We validated Lingo with objective measures of language ability by comparing performance to gold-standard assessments: CELF-5 and the VABS-3. Factor 2 was significantly associated with the CELF-5 "core language ability" scale (Pearson's R = 0.77, p-value < 0.05) and the VABS-3 "communication" scale (Pearson's R = 0.74, p value < 0.05). Factor 2 was positively associated with phenotypic and genetic measures of socieconomic status. Interestingly, we found the parents of children with language impairments had lower Factor 2 scores (p-value < 0.01). Finally, we found Lingo factor scores were significantly predictive of numerous psychiatric and neurodevelopmental conditions. Conclusions: Together, these analyses support Lingo as a powerful platform for scalable deep phenotyping of language and other cognitive abilities. Additionally, exploratory analyses provide supporting evidence for the heritability of language ability and the complex relationship between mental health and language.

On measuring linguistic intelligence

Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages

Linguini: A benchmark for language-agnostic linguistic reasoning

Quantifying AI Psychology: A Psychometrics Benchmark for Large Language Models

Linguistic Features Distinguishing Students' Writing Ability Aligned with CEFR Levels

How to Measure the Intelligence of Large Language Models?

Lingo: an automated, web-based deep phenotyping platform for language ability

Measurement in the Age of LLMs: An Application to Ideological Scaling

Probabilistic Method of Measuring Linguistic Productivity

On a measure of intelligence

New Unbalanced Linguistic Scale Sets: the Linguistic Information Representations and Applications

From traces to measures: Large language models as a tool for psychological measurement from text

Quantifying the Dialect Gap and its Correlates Across Languages

PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Models

A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets

Human Behavioral Benchmarking: Numeric Magnitude Comparison Effects in Large Language Models

Linguistic Diversity Index: A Scientometric Measure to Enhance the Relevance of Small and Minority Group Languages

Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models

A machine learning-based investigation of syntactic complexity measures in second language acquisition of Chinese

Measures of lexical distance between languages

Quantifying Intimacy in Language