Abstract:The Artificial Intelligence (AI) research community has used ad-hoc benchmarks to measure the "intelligence" level of Large Language Models (LLMs). In humans, intelligence is closely linked to the functional integrity of the prefrontal lobes, which are essential for higher-order cognitive processes. Previous research has found that LLMs struggle with cognitive tasks that rely on these prefrontal functions, highlighting a significant challenge in replicating human-like intelligence. In December 2022, OpenAI released ChatGPT, a new chatbot based on the GPT-3.5 model that quickly gained popularity for its impressive ability to understand and respond to human instructions, suggesting a significant step towards intelligent behaviour in AI. Therefore, to rigorously investigate LLMs' level of "intelligence," we evaluated the GPT-3.5 and GPT-4 versions through a neuropsychological assessment using tests in the Italian language routinely employed to assess prefrontal functioning in humans. The same tests were also administered to Claude2 and Llama2 to verify whether similar language models perform similarly in prefrontal tests. When using human performance as a reference, GPT-3.5 showed inhomogeneous results on prefrontal tests, with some tests well above average, others in the lower range, and others frankly impaired. Specifically, we have identified poor planning abilities and difficulty in recognising semantic absurdities and understanding others' intentions and mental states. Claude2 exhibited a similar pattern to GPT-3.5, while Llama2 performed poorly in almost all tests. These inconsistent profiles highlight how LLMs' emergent abilities do not yet mimic human cognitive functioning. The sole exception was GPT-4, which performed within the normative range for all the tasks except planning. Furthermore, we showed how standardised neuropsychological batteries developed to assess human cognitive functions may be suitable for challenging LLMs' performance.

Neural Generative Models and the Parallel Architecture of Language: A Critical Review and Outlook

A Philosophical Introduction to Language Models -- Part I: Continuity With Classic Debates

Towards a neural architecture of language: Deep learning versus logistics of access in neural architectures for compositional processing

A generative framework to bridge data-driven models and scientific theories in language neuroscience

Neural Language Generation: Formulation, Methods, and Evaluation

Multiversal views on language models

Generative AI for Complex Scenarios: Language Models Are Sequence Processors

Cutting Through the Confusion and Hype: Understanding the True Potential of Generative AI

Fundamentals of Generative Large Language Models and Perspectives in Cyber-Defense

Large language models (LLMs): survey, technical frameworks, and future challenges

Artificial Neuropsychology: Are Large Language Models Developing Executive Functions?

Large Language Models and Generative AI, Oh My!

Large Linguistic Models: Analyzing theoretical linguistic abilities of LLMs

Challenging large language models' " intelligence" with human tools: A neuropsychological investigation in Italian language on prefrontal functioning

A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Hum4n L4ngu4ge and the W0rld behind W0rds?

Can Large Language Models Write Parallel Code?

Generative Large Language Models Explained [Ai-Explained]

Evaluating Neural Language Models as Cognitive Models of Language Acquisition

Beyond Generative Artificial Intelligence: Roadmap for Natural Language Generation

Linguistic generalization and compositionality in modern artificial neural networks

Generative AI in the Era of Transformers: Revolutionizing Natural Language Processing with LLMs