Abstract:Developmental psychologists have spent decades devising experiments to test the intelligence and knowledge of infants and children, tracing the origin of crucial concepts and capacities. Moreover, experimental techniques in developmental psychology have been carefully designed to discriminate the cognitive capacities that underlie particular behaviors. We propose that using classical experiments from child development is a particularly effective way to probe the computational abilities of AI models, in general, and LLMs in particular. First, the methodological techniques of developmental psychology, such as the use of novel stimuli to control for past experience or control conditions to determine whether children are using simple associations, can be equally helpful for assessing the capacities of LLMs. In parallel, testing LLMs in this way can tell us whether the information that is encoded in text is sufficient to enable particular responses, or whether those responses depend on other kinds of information, such as information from exploration of the physical world. In this work we adapt classical developmental experiments to evaluate the capabilities of LaMDA, a large language model from Google. We propose a novel LLM Response Score (LRS) metric which can be used to evaluate other language models, such as GPT. We find that LaMDA generates appropriate responses that are similar to those of children in experiments involving social understanding, perhaps providing evidence that knowledge of these domains is discovered through language. On the other hand, LaMDA's responses in early object and action understanding, theory of mind, and especially causal reasoning tasks are very different from those of young children, perhaps showing that these domains require more real-world, self-initiated exploration and cannot simply be learned from patterns in language input.

"What is on your mind?" Automated Scoring of Mindreading in Childhood and Early Adolescence

Measure Children’s Mindreading Ability with Machine Reading

Children's On-Line Processing of Epistemic Modals.

Machine learning and deep learning systems for automated measurement of "advanced" theory of mind: Reliability and validity in children and adolescents.

Development and Validation of a Mobile Application Assessing Young Children’s Social Cognition Abilities

Comparing Machines and Children: Using Developmental Psychology Experiments to Assess the Strengths and Weaknesses of LaMDA Responses

Examining interpersonal metacognitive monitoring in artificial grammar learning

Mindreading quality versus quantity: A theoretically and empirically motivated two-factor structure for individual differences in adults' mindreading

An Initial Investigation of Non-Native Spoken Question-Answering

Who Said What? An Automated Approach to Analyzing Speech in Preschool Classrooms

MMToM-QA: Multimodal Theory of Mind Question Answering

MindScope: Exploring cognitive biases in large language models through Multi-Agent Systems

Evaluating the accuracy of automated processing of child and adult language production in preschool classrooms

Through the Theory of Mind's Eye: Reading Minds with Multimodal Video Large Language Models

GPT-4o reads the mind in the eyes

Human Behavior Inspired Machine Reading Comprehension

Automatic Analysis of the Emotional Content of Speech in Daylong Child-Centered Recordings from a Neonatal Intensive Care Unit

Smarnet: Teaching Machines to Read and Comprehend Like Human.

Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation

Towards Social & Engaging Peer Learning: Predicting Backchanneling and Disengagement in Children

An Enactivist account of Mind Reading in Natural Language Understanding