Beyond Turing: Testing LLMs for Intelligence

Neil Savage

DOI: https://doi.org/10.1145/3673427

IF: 22.7

2024-08-27

Communications of the ACM

Abstract:Computer scientists have not developed a consensus on how to measure intelligence, or what exactly to measure.

computer science, theory & methods, software engineering, hardware & architecture

What problem does this paper attempt to address?

The Turing Test and our shifting conceptions of intelligence

Melanie Mitchell

DOI: https://doi.org/10.1126/science.adq9356

IF: 56.9

2024-08-16

Science

Abstract:"Can machines think?" So asked Alan Turing in his 1950 paper, "Computing Machinery and Intelligence." Turing quickly noted that, given the difficulty of defining thinking, the question is "too meaningless to deserve discussion." As is often done in philosophical debates, he proposed replacing it with a different question. Turing imagined an "imitation game," in which a human judge converses with both a computer and a human (a "foil"), each of which vies to convince the judge that they are the human. Importantly, the computer, foil, and judge do not see one another; they communicate entirely through text. After conversing with each candidate, the judge guesses which one is the real human. Turing's new question was, "Are there imaginable digital computers which would do well in the imitation game?"
Tests of Machine Intelligence

Shane Legg,Marcus Hutter

DOI: https://doi.org/10.48550/arXiv.0712.3825

IF: 14.4

2007-12-22

Artificial Intelligence

Abstract:Although the definition and measurement of intelligence is clearly of fundamental importance to the field of artificial intelligence, no general survey of definitions and tests of machine intelligence exists. Indeed few researchers are even aware of alternatives to the Turing test and its many derivatives. In this paper we fill this gap by providing a short survey of the many tests of machine intelligence that have been proposed.
Intelligence as a Social Concept: a Socio-Technological Interpretation of the Turing Test

Shlomo Danziger

DOI: https://doi.org/10.1007/s13347-022-00561-z

2022-07-24

Abstract:Alan Turing's 1950 imitation game has been widely understood as a means for testing if an entity is intelligent. Following a series of papers by Diane Proudfoot, I offer a socio-technological interpretation of Turing's paper and present an alternative way of understanding both the imitation game and Turing's concept of intelligence. Turing, I claim, saw intelligence as a social concept, meaning that possession of intelligence is a property determined by society's attitude toward the entity. He realized that as long as human society held a prejudiced attitude toward machinery—seeing machines a priori as mindless objects—machines could not be said to be intelligent, by definition. He also realized, though, that if humans' a priori, chauvinistic attitude toward machinery changed, the existence of intelligent machines would become logically possible. Turing thought that such a change would eventually occur: He believed that when scientists overcome the technological challenge of constructing sophisticated machines that could imitate human verbal behavior—i.e., do well in the imitation game—humans' prejudiced attitude toward machinery will have altered in such a way that machines could be said to be intelligent. The imitation game, for Turing, was not an intelligence test, but a technological aspiration whose realization would likely involve a change in society's attitude toward machines.
How to Measure the Intelligence of Large Language Models?

Nils Körber,Silvan Wehrli,Christopher Irrgang

2024-07-30

Abstract:With the release of ChatGPT and other large language models (LLMs) the discussion about the intelligence, possibilities, and risks, of current and future models have seen large attention. This discussion included much debated scenarios about the imminent rise of so-called "super-human" AI, i.e., AI systems that are orders of magnitude smarter than humans. In the spirit of Alan Turing, there is no doubt that current state-of-the-art language models already pass his famous test. Moreover, current models outperform humans in several benchmark tests, so that publicly available LLMs have already become versatile companions that connect everyday life, industry and science. Despite their impressive capabilities, LLMs sometimes fail completely at tasks that are thought to be trivial for humans. In other cases, the trustworthiness of LLMs becomes much more elusive and difficult to evaluate. Taking the example of academia, language models are capable of writing convincing research articles on a given topic with only little input. Yet, the lack of trustworthiness in terms of factual consistency or the existence of persistent hallucinations in AI-generated text bodies has led to a range of restrictions for AI-based content in many scientific journals. In view of these observations, the question arises as to whether the same metrics that apply to human intelligence can also be applied to computational methods and has been discussed extensively. In fact, the choice of metrics has already been shown to dramatically influence assessments on potential intelligence emergence. Here, we argue that the intelligence of LLMs should not only be assessed by task-specific statistical metrics, but separately in terms of qualitative and quantitative measures.

Artificial Intelligence,Machine Learning
Learning, Social Intelligence and the Turing Test - why an "out-of-the-box" Turing Machine will not pass the Turing Test

Bruce Edmonds,Carlos Gershenson

DOI: https://doi.org/10.48550/arXiv.1203.3376

IF: 14.4

2012-03-15

Artificial Intelligence

Abstract:The Turing Test (TT) checks for human intelligence, rather than any putative general intelligence. It involves repeated interaction requiring learning in the form of adaption to the human conversation partner. It is a macro-level post-hoc test in contrast to the definition of a Turing Machine (TM), which is a prior micro-level definition. This raises the question of whether learning is just another computational process, i.e. can be implemented as a TM. Here we argue that learning or adaption is fundamentally different from computation, though it does involve processes that can be seen as computations. To illustrate this difference we compare (a) designing a TM and (b) learning a TM, defining them for the purpose of the argument. We show that there is a well-defined sequence of problems which are not effectively designable but are learnable, in the form of the bounded halting problem. Some characteristics of human intelligence are reviewed including it's: interactive nature, learning abilities, imitative tendencies, linguistic ability and context-dependency. A story that explains some of these is the Social Intelligence Hypothesis. If this is broadly correct, this points to the necessity of a considerable period of acculturation (social learning in context) if an artificial intelligence is to pass the TT. Whilst it is always possible to 'compile' the results of learning into a TM, this would not be a designed TM and would not be able to continually adapt (pass future TTs). We conclude three things, namely that: a purely "designed" TM will never pass the TT; that there is no such thing as a general intelligence since it necessary involves learning; and that learning/adaption and computation should be clearly distinguished.
Turing Test Revisited: A Framework for an Alternative

Aladdin Ayesh

DOI: https://doi.org/10.48550/arXiv.1906.11068

IF: 14.4

2019-06-26

Artificial Intelligence

Abstract:This paper aims to question the suitability of the Turing Test, for testing machine intelligence, in the light of advances made in the last 60 years in science, medicine, and philosophy of mind. While the main concept of the test may seem sound and valid, a detailed analysis of what is required to pass the test highlights a significant flow. Once the analysis of the test is presented, a systematic approach is followed in analysing what is needed to devise a test or tests for intelligent machines. The paper presents a plausible generic framework based on categories of factors implied by subjective perception of intelligence. An evaluative discussion concludes the paper highlighting some of the unaddressed issues within this generic framework.
Deceptive Appearances: the Turing Test, Response-Dependence, and Intelligence as an Emotional Concept

Michael Wheeler

DOI: https://doi.org/10.1007/s11023-020-09533-8

IF: 5.339

2020-08-12

Minds and Machines

Abstract:Abstract The Turing Test is routinely understood as a behaviourist test for machine intelligence. Diane Proudfoot (Rethinking Turing’s Test, Journal of Philosophy , 2013) has argued for an alternative interpretation. According to Proudfoot, Turing’s claim that intelligence is what he calls ‘an emotional concept’ indicates that he conceived of intelligence in response-dependence terms. As she puts it: ‘Turing’s criterion for “thinking” is...: x is intelligent (or thinks) if in the actual world, in an unrestricted computer-imitates-human game, x appears intelligent to an average interrogator’. The role of the famous test is thus to provide the conditions in which to examine the average interrogator’s responses. I shall argue that Proudfoot’s analysis falls short. The philosophical literature contains two main models of response-dependence, what I shall call the transparency model and the reference - fixing model. Proudfoot resists the thought that Turing might have endorsed one of these models to the exclusion of the other. But the details of her own analysis indicate that she is, in fact , committed to the claim that Turing’s account of intelligence is grounded in a transparency model, rather than a reference-fixing one. By contrast, I shall argue that while Turing did indeed conceive of intelligence in response-dependence terms, his account is grounded in a reference-fixing model, rather than a transparency one. This is fortunate (for Turing), because, as an account of intelligence, the transparency model is arguably problematic in a way that the reference-fixing model isn’t.

computer science, artificial intelligence
Post Turing: Mapping the landscape of LLM Evaluation

Alexey Tikhonov,Ivan P. Yamshchikov

2023-11-04

Abstract:In the rapidly evolving landscape of Large Language Models (LLMs), introduction of well-defined and standardized evaluation methodologies remains a crucial challenge. This paper traces the historical trajectory of LLM evaluations, from the foundational questions posed by Alan Turing to the modern era of AI research. We categorize the evolution of LLMs into distinct periods, each characterized by its unique benchmarks and evaluation criteria. As LLMs increasingly mimic human-like behaviors, traditional evaluation proxies, such as the Turing test, have become less reliable. We emphasize the pressing need for a unified evaluation system, given the broader societal implications of these models. Through an analysis of common evaluation methodologies, we advocate for a qualitative shift in assessment approaches, underscoring the importance of standardization and objective criteria. This work serves as a call for the AI community to collaboratively address the challenges of LLM evaluation, ensuring their reliability, fairness, and societal benefit.

Computation and Language,Artificial Intelligence
Measuring Intelligence through Games

Tom Schaul,Julian Togelius,Jürgen Schmidhuber

DOI: https://doi.org/10.48550/arXiv.1109.1314

2011-09-07

Abstract:Artificial general intelligence (AGI) refers to research aimed at tackling the full problem of artificial intelligence, that is, create truly intelligent agents. This sets it apart from most AI research which aims at solving relatively narrow domains, such as character recognition, motion planning, or increasing player satisfaction in games. But how do we know when an agent is truly intelligent? A common point of reference in the AGI community is Legg and Hutter's formal definition of universal intelligence, which has the appeal of simplicity and generality but is unfortunately incomputable. Games of various kinds are commonly used as benchmarks for "narrow" AI research, as they are considered to have many important properties. We argue that many of these properties carry over to the testing of general intelligence as well. We then sketch how such testing could practically be carried out. The central part of this sketch is an extension of universal intelligence to deal with finite time, and the use of sampling of the space of games expressed in a suitably biased game description language.

Artificial Intelligence
Beyond Interpretable Benchmarks: Contextual Learning through Cognitive and Multimodal Perception

Nick DiSanto

2023-09-30

Abstract:With state-of-the-art models achieving high performance on standard benchmarks, contemporary research paradigms continue to emphasize general intelligence as an enduring objective. However, this pursuit overlooks the fundamental disparities between the high-level data perception abilities of artificial and natural intelligence systems. This study questions the Turing Test as a criterion of generally intelligent thought and contends that it is misinterpreted as an attempt to anthropomorphize computer systems. Instead, it emphasizes tacit learning as a cornerstone of general-purpose intelligence, despite its lack of overt interpretability. This abstract form of intelligence necessitates contextual cognitive attributes that are crucial for human-level perception: generalizable experience, moral responsibility, and implicit prioritization. The absence of these features yields undeniable perceptual disparities and constrains the cognitive capacity of artificial systems to effectively contextualize their environments. Additionally, this study establishes that, despite extensive exploration of potential architecture for future systems, little consideration has been given to how such models will continuously absorb and adapt to contextual data. While conventional models may continue to improve in benchmark performance, disregarding these contextual considerations will lead to stagnation in human-like comprehension. Until general intelligence can be abstracted from task-specific domains and systems can learn implicitly from their environments, research standards should instead prioritize the disciplines in which AI thrives.

Artificial Intelligence
On the Measure of Intelligence

François Chollet

DOI: https://doi.org/10.48550/arXiv.1911.01547

2019-11-25

Abstract:To make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans. Over the past hundred years, there has been an abundance of attempts to define and measure intelligence, across both the fields of psychology and AI. We summarize and critically assess these definitions and evaluation approaches, while making apparent the two historical conceptions of intelligence that have implicitly guided them. We note that in practice, the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs and humans at specific tasks such as board games and video games. We argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to "buy" arbitrary levels of skills for a system, in a way that masks the system's own generalization power. We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. Using this definition, we propose a set of guidelines for what a general AI benchmark should look like. Finally, we present a benchmark closely following these guidelines, the Abstraction and Reasoning Corpus (ARC), built upon an explicit set of priors designed to be as close as possible to innate human priors. We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.

Artificial Intelligence
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence

Timothy R. McIntosh,Teo Susnjak,Tong Liu,Paul Watters,Malka N. Halgamuge

DOI: https://doi.org/10.48550/arXiv.2402.09880

IF: 14.4

2024-02-15

Artificial Intelligence

Abstract:The rapid rise in popularity of Large Language Models (LLMs) with emerging capabilities has spurred public curiosity to evaluate and compare different LLMs, leading many researchers to propose their LLM benchmarks. Noticing preliminary inadequacies in those benchmarks, we embarked on a study to critically assess 23 state-of-the-art LLM benchmarks, using our novel unified evaluation framework through the lenses of people, process, and technology, under the pillars of functionality and security. Our research uncovered significant limitations, including biases, difficulties in measuring genuine reasoning, adaptability, implementation inconsistencies, prompt engineering complexity, evaluator diversity, and the overlooking of cultural and ideological norms in one comprehensive assessment. Our discussions emphasized the urgent need for standardized methodologies, regulatory certainties, and ethical guidelines in light of Artificial Intelligence (AI) advancements, including advocating for an evolution from static benchmarks to dynamic behavioral profiling to accurately capture LLMs' complex behaviors and potential risks. Our study highlighted the necessity for a paradigm shift in LLM evaluation methodologies, underlining the importance of collaborative efforts for the development of universally accepted benchmarks and the enhancement of AI systems' integration into society.
"Turing Tests" For An AI Scientist

Xiaoxin Yin

2024-05-22

Abstract:While LLMs have shown impressive capabilities in solving math or coding problems, the ability to make scientific discoveries remains a distinct challenge. This paper proposes a "Turing test for an AI scientist" to assess whether an AI agent can conduct scientific research independently, without relying on human-generated knowledge. Drawing inspiration from the historical development of science, we propose seven benchmark tests that evaluate an AI agent's ability to make groundbreaking discoveries in various scientific domains. These tests include inferring the heliocentric model from celestial observations, discovering the laws of motion in a simulated environment, deriving the differential equation governing vibrating strings, inferring Maxwell's equations from electrodynamics simulations, inventing numerical methods for initial value problems, discovering Huffman coding for data compression, and developing efficient sorting algorithms. To ensure the validity of these tests, the AI agent is provided with interactive libraries or datasets specific to each problem, without access to human knowledge that could potentially contain information about the target discoveries. The ultimate goal is to create an AI scientist capable of making novel and impactful scientific discoveries, surpassing the best human experts in their respective fields. These "Turing tests" serve as intermediate milestones, assessing the AI agent's ability to make discoveries that were groundbreaking in their time. If an AI agent can pass the majority of these seven tests, it would indicate significant progress towards building an AI scientist, paving the way for future advancements in autonomous scientific discovery. This paper aims to establish a benchmark for the capabilities of AI in scientific research and to stimulate further research in this exciting field.

Artificial Intelligence
On a measure of intelligence

Yuri Gurevich

2024-09-22

Abstract:The Fall 2024 Logic in Computer Science column of the Bulletin of EATCS is a little discussion on intelligence, measuring intelligence, and related issues, provoked by a fascinating must-read article ``On the measure of intelligence'' by François Chollet. The discussion includes a modicum of critique of the article.

Artificial Intelligence
The Moral Turing Test: Evaluating Human-LLM Alignment in Moral Decision-Making

Basile Garcia,Crystal Qian,Stefano Palminteri

2024-10-10

Abstract:As large language models (LLMs) become increasingly integrated into society, their alignment with human morals is crucial. To better understand this alignment, we created a large corpus of human- and LLM-generated responses to various moral scenarios. We found a misalignment between human and LLM moral assessments; although both LLMs and humans tended to reject morally complex utilitarian dilemmas, LLMs were more sensitive to personal framing. We then conducted a quantitative user study involving 230 participants (N=230), who evaluated these responses by determining whether they were AI-generated and assessed their agreement with the responses. Human evaluators preferred LLMs' assessments in moral scenarios, though a systematic anti-AI bias was observed: participants were less likely to agree with judgments they believed to be machine-generated. Statistical and NLP-based analyses revealed subtle linguistic differences in responses, influencing detection and agreement. Overall, our findings highlight the complexities of human-AI perception in morally charged decision-making.

Human-Computer Interaction,Artificial Intelligence
Can machines think? The controversy that led to the Turing test

Bernardo Gonçalves

DOI: https://doi.org/10.1007/s00146-021-01318-6

2022-01-11

Abstract:Turing’s much debated test has turned 70 and is still fairly controversial. His 1950 paper is seen as a complex and multilayered text, and key questions about it remain largely unanswered. Why did Turing select learning from experience as the best approach to achieve machine intelligence? Why did he spend several years working with chess playing as a task to illustrate and test for machine intelligence only to trade it out for conversational question-answering in 1950? Why did Turing refer to gender imitation in a test for machine intelligence? In this article, I shall address these questions by unveiling social, historical and epistemological roots of the so-called Turing test. I will draw attention to a historical fact that has been only scarcely observed in the secondary literature thus far, namely that Turing’s 1950 test emerged out of a controversy over the cognitive capabilities of digital computers, most notably out of debates with physicist and computer pioneer Douglas Hartree, chemist and philosopher Michael Polanyi, and neurosurgeon Geoffrey Jefferson. Seen in its historical context, Turing’s 1950 paper can be understood as essentially a reply to a series of challenges posed to him by these thinkers arguing against his view that machines can think. Turing did propose gender learning and imitation as one of his various imitation tests for machine intelligence, and I argue here that this was done in response to Jefferson's suggestion that gendered behavior is causally related to the physiology of sex hormones.
The Darmstadt Challenge - the Turing Test Revisited

J. C. Augusto,M. Bohlen,D. Cook,F. Flentge,G. Marreiros,Carlos Ramos,Weijun Qin,Yue Suo

DOI: https://doi.org/10.5220/0001665002910296

2009-01-01

Abstract:Significant work has been done in the areas of Pervcomp/Ubicomp/Smart Environments with advances on making proactive systems, but those advances have not made these type of systems accurately proactive. On the other hand a great deal is needed to make systems more sensible/sensitive and trustable (both in terms of reliability and privacy). We put forward the thesis that a more integral and social-aware sort of intelligence is needed to effectively interact, decide and act on behalf of people’s interest and that a way to test how effective systems are achieving these desirable behaviour is needed as a consequence. We support our thesis by providing examples on how to measure effectiveness in variety of different environments.
A Survey of Question Answering for Math and Science Problem

Arindam Bhattacharya

DOI: https://doi.org/10.48550/arXiv.1705.04530

IF: 14.4

2017-05-10

Artificial Intelligence

Abstract:Turing test was long considered the measure for artificial intelligence. But with the advances in AI, it has proved to be insufficient measure. We can now aim to mea- sure machine intelligence like we measure human intelligence. One of the widely accepted measure of intelligence is standardized math and science test. In this paper, we explore the progress we have made towards the goal of making a machine smart enough to pass the standardized test. We see the challenges and opportunities posed by the domain, and note that we are quite some ways from actually making a system as smart as a even a middle school scholar.
TuringQ: Benchmarking AI Comprehension in Theory of Computation

Pardis Sadat Zahraei,Ehsaneddin Asgari

2024-10-09

Abstract:We present TuringQ, the first benchmark designed to evaluate the reasoning capabilities of large language models (LLMs) in the theory of computation. TuringQ consists of 4,006 undergraduate and graduate-level question-answer pairs, categorized into four difficulty levels and covering seven core theoretical areas. We evaluate several open-source LLMs, as well as GPT-4, using Chain of Thought prompting and expert human assessment. Additionally, we propose an automated LLM-based evaluation system that demonstrates competitive accuracy when compared to human evaluation. Fine-tuning a Llama3-8B model on TuringQ shows measurable improvements in reasoning ability and out-of-domain tasks such as algebra. TuringQ serves as both a benchmark and a resource for enhancing LLM performance in complex computational reasoning tasks. Our analysis offers insights into LLM capabilities and advances in AI comprehension of theoretical computer science.

Computation and Language,Formal Languages and Automata Theory

Beyond Turing: Testing LLMs for Intelligence

The Turing Test and our shifting conceptions of intelligence

Tests of Machine Intelligence

Intelligence as a Social Concept: a Socio-Technological Interpretation of the Turing Test

How to Measure the Intelligence of Large Language Models?

Learning, Social Intelligence and the Turing Test - why an "out-of-the-box" Turing Machine will not pass the Turing Test

Turing Test Revisited: A Framework for an Alternative

Deceptive Appearances: the Turing Test, Response-Dependence, and Intelligence as an Emotional Concept

Post Turing: Mapping the landscape of LLM Evaluation

Measuring Intelligence through Games

Beyond Interpretable Benchmarks: Contextual Learning through Cognitive and Multimodal Perception

On the Measure of Intelligence

Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence

"Turing Tests" For An AI Scientist

On a measure of intelligence

The Moral Turing Test: Evaluating Human-LLM Alignment in Moral Decision-Making

Can machines think? The controversy that led to the Turing test

The Darmstadt Challenge - the Turing Test Revisited

A Survey of Question Answering for Math and Science Problem

TuringQ: Benchmarking AI Comprehension in Theory of Computation