Abstract:Large language models (LLMs) have shown remarkable abilities recently, including passing advanced professional exams and demanding benchmark tests. This performance has led many to suggest that they are close to achieving humanlike or "true" understanding of language, and even artificial general intelligence (AGI). Here, we provide a new open-source benchmark, the Two Word Test (TWT), that can assess semantic abilities of LLMs using two-word phrases in a task that can be performed relatively easily by humans without advanced training. Combining multiple words into a single concept is a fundamental linguistic and conceptual operation routinely performed by people. The test requires meaningfulness judgments of 1768 noun-noun combinations that have been rated as meaningful (e.g., baby boy) or as having low meaningfulness (e.g., goat sky) by human raters. This novel test differs from existing benchmarks that rely on logical reasoning, inference, puzzle-solving, or domain expertise. We provide versions of the task that probe meaningfulness ratings on a 0-4 scale as well as binary judgments. With both versions, we conducted a series of experiments using the TWT on GPT-4, GPT-3.5, Claude-3-Optus, and Gemini-1-Pro-001. Results demonstrated that, compared to humans, all models performed relatively poorly at rating meaningfulness of these phrases. GPT-3.5-turbo, Gemini-1.0-Pro-001 and GPT-4-turbo were also unable to make binary discriminations between sensible and nonsense phrases, with these models consistently judging nonsensical phrases as making sense. Claude-3-Opus made a substantial improvement in binary discrimination of combinatorial phrases but was still significantly worse than human performance. The TWT can be used to understand and assess the limitations of current LLMs, and potentially improve them. The test also reminds us that caution is warranted in attributing "true" or human-level understanding to LLMs based only on tests that are challenging for humans.

Word Clarity As a Metric in Sampling Keyboard Test Sets.

A data quality improvement method based on non-word errors correction

Scoring Metrics of Assessing Voiceprint Distinctiveness Based on Speech Content and Rate

An Automated Evaluation Metric for Chinese Text Entry

The Base-Rate Effect on LLM Benchmark Performance: Disambiguating Test-Taking Strategies from Benchmark Performance

Typing Efficiency and Suggestion Accuracy Influence the Benefits and Adoption of Word Suggestions

The Web-based Open-access Reliable Decision on Synonyms (WORDS) English Vocabulary Test

Performance of Skilled Typists As a Function of Pacing, Response Complexity and Urgency in Numerical Hear-and-type Tasks

The Two Word Test as a semantic benchmark for large language models

Is it too small?: Investigating the performances and preferences of users when typing on tiny QWERTY keyboards.

The Role of Entropy in Construct Specification Equations (CSE) to Improve the Validity of Memory Tests: Extension to Word Lists

Optimizing Cognitive Assessment Outcome Measures for Alzheimer's Disease by Matching Wordlist Memory Test Features to Scoring Methodology

Adaptive Approach to Concordance Readability

Are Some Words Worth More than Others?

Smart Word Suggestions for Writing Assistance

Moving beyond Kucera and Francis: a critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English

Word Forms Reflect Trade-Offs Between Speaker Effort and Robust Listener Recognition

Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies

CLiMP: A Benchmark for Chinese Language Model Evaluation

CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models

Factors affecting numerical typing performance of young adults in a hear-and-type task.