Investigating Context Effects in Similarity Judgements in Large Language Models

Sagar Uprety,Amit Kumar Jaiswal,Haiming Liu,Dawei Song

2024-08-20

Abstract:Large Language Models (LLMs) have revolutionised the capability of AI models in comprehending and generating natural language text. They are increasingly being used to empower and deploy agents in real-world scenarios, which make decisions and take actions based on their understanding of the context. Therefore researchers, policy makers and enterprises alike are working towards ensuring that the decisions made by these agents align with human values and user expectations. That being said, human values and decisions are not always straightforward to measure and are subject to different cognitive biases. There is a vast section of literature in Behavioural Science which studies biases in human judgements. In this work we report an ongoing investigation on alignment of LLMs with human judgements affected by order bias. Specifically, we focus on a famous human study which showed evidence of order effects in similarity judgements, and replicate it with various popular LLMs. We report the different settings where LLMs exhibit human-like order effect bias and discuss the implications of these findings to inform the design and development of LLM based applications.

Artificial Intelligence

What problem does this paper attempt to address?

The problem this paper attempts to address is whether large language models (LLMs) exhibit order effect biases similar to humans in similarity judgments. Specifically, the authors replicate a famous study by Tversky and Gati, which demonstrated the presence of order effects in human similarity judgments, meaning that the order in which objects are presented affects the judgment of their similarity. The authors repeat this experiment using various popular LLMs to explore whether these models also exhibit similar human-like order effect biases under different settings and discuss the implications of these findings for the design and development of LLM-based applications. The main research questions of the paper are: - To what extent do large language models align with human judgments in context-sensitive similarity assessments? - Under what conditions do LLMs exhibit order effect biases similar to those of humans?

Investigating Context Effects in Similarity Judgements in Large Language Models

Investigating Implicit Bias in Large Language Models: A Large-Scale Study of Over 50 LLMs

Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challenges

Bias Similarity Across Large Language Models

Exploring Value Biases: How LLMs Deviate Towards the Ideal

Humanlike Cognitive Patterns as Emergent Phenomena in Large Language Models

Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Models

Aligning with Logic: Measuring, Evaluating and Improving Logical Consistency in Large Language Models

Cognitive Bias in Decision-Making with LLMs

CBEval: A framework for evaluating and interpreting cognitive biases in LLMs

People's Perceptions Toward Bias and Related Concepts in Large Language Models: A Systematic Review

Language models and psychological sciences

Exploring Accuracy-Fairness Trade-off in Large Language Models

Large Language Models Show Human-like Social Desirability Biases in Survey Responses

Improving Large Language Model (LLM) fidelity through context-aware grounding: A systematic approach to reliability and veracity

Mind vs. Mouth: On Measuring Re-judge Inconsistency of Social Bias in Large Language Models

MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks

Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators

Are Large Language Models Chameleons? An Attempt to Simulate Social Surveys

Contextual Feature Extraction Hierarchies Converge in Large Language Models and the Brain