Abstract:Large language models (LLMs) have shown remarkable abilities recently, including passing advanced professional exams and demanding benchmark tests. This performance has led many to suggest that they are close to achieving humanlike or "true" understanding of language, and even artificial general intelligence (AGI). Here, we provide a new open-source benchmark, the Two Word Test (TWT), that can assess semantic abilities of LLMs using two-word phrases in a task that can be performed relatively easily by humans without advanced training. Combining multiple words into a single concept is a fundamental linguistic and conceptual operation routinely performed by people. The test requires meaningfulness judgments of 1768 noun-noun combinations that have been rated as meaningful (e.g., baby boy) or as having low meaningfulness (e.g., goat sky) by human raters. This novel test differs from existing benchmarks that rely on logical reasoning, inference, puzzle-solving, or domain expertise. We provide versions of the task that probe meaningfulness ratings on a 0-4 scale as well as binary judgments. With both versions, we conducted a series of experiments using the TWT on GPT-4, GPT-3.5, Claude-3-Optus, and Gemini-1-Pro-001. Results demonstrated that, compared to humans, all models performed relatively poorly at rating meaningfulness of these phrases. GPT-3.5-turbo, Gemini-1.0-Pro-001 and GPT-4-turbo were also unable to make binary discriminations between sensible and nonsense phrases, with these models consistently judging nonsensical phrases as making sense. Claude-3-Opus made a substantial improvement in binary discrimination of combinatorial phrases but was still significantly worse than human performance. The TWT can be used to understand and assess the limitations of current LLMs, and potentially improve them. The test also reminds us that caution is warranted in attributing "true" or human-level understanding to LLMs based only on tests that are challenging for humans.

How operationalizations of word types affect measures of lexical diversity

Assessing the Validity of Lexical Diversity Indices Using Direct Judgements

Investigating minimum text lengths for lexical diversity indices

Lexical Diversity As a Lens into the Classification of Slavic Languages: A Quantitative Typology Perspective.

Lexical Category Bias Across Interpreting Types: Implications for Synergy Between Cognitive Constraints and Language Representations

Should We Stop Using Lexical Diversity Measures in Children's Language Sample Analysis?

A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets

Coming to grips with lexical richness in spontaneous speech data

What is "Typological Diversity" in NLP?

EVALUATING L2 VOCABULARY DEVELOPMENT FEATURES USING LEXICAL DENSITY AND LEXICAL DIVERSITY MEASURES

Measuring Lexical Diversity in Texts: The Twofold Length Problem

Entropy and type-token ratio in gigaword corpora

Locally Measuring Cross-lingual Lexical Alignment: A Domain and Word Level Perspective

Lexical Diversity Versus Lexical Error in the Language Transcripts of Children With Developmental Language Disorder: Different Conclusions About Lexical Ability.

The LSCD Benchmark: a Testbed for Diachronic Word Meaning Tasks

Spontane Taaldata En Het Meten Van Lexicale Rijkdom In Tweedetaalverwerving

Benchmarking Linguistic Diversity of Large Language Models

The Two Word Test as a semantic benchmark for large language models

Lexical Richness in the Spontaneous Speech of Bilinguals

Reliability of lexical richness measures based on word lists in short second language productions

Measuring Grammatical Diversity from Small Corpora: Derivational Entropy Rates, Mean Length of Utterances, and Annotation Invariance