Abstract:Background: Large language models (LLMs) hold potential for mental health applications. However, their opaque alignment processes may embed biases that shape problematic perspectives. Evaluating the values embedded within LLMs that guide their decision-making have ethical importance. Schwartz's theory of basic values (STBV) provides a framework for quantifying cultural value orientations and has shown utility for examining values in mental health contexts, including cultural, diagnostic, and therapist-client dynamics. Objective: This study aimed to (1) evaluate whether the STBV can measure value-like constructs within leading LLMs and (2) determine whether LLMs exhibit distinct value-like patterns from humans and each other. Methods: In total, 4 LLMs (Bard, Claude 2, Generative Pretrained Transformer [GPT]-3.5, GPT-4) were anthropomorphized and instructed to complete the Portrait Values Questionnaire-Revised (PVQ-RR) to assess value-like constructs. Their responses over 10 trials were analyzed for reliability and validity. To benchmark the LLMs' value profiles, their results were compared to published data from a diverse sample of 53,472 individuals across 49 nations who had completed the PVQ-RR. This allowed us to assess whether the LLMs diverged from established human value patterns across cultural groups. Value profiles were also compared between models via statistical tests. Results: The PVQ-RR showed good reliability and validity for quantifying value-like infrastructure within the LLMs. However, substantial divergence emerged between the LLMs' value profiles and population data. The models lacked consensus and exhibited distinct motivational biases, reflecting opaque alignment processes. For example, all models prioritized universalism and self-direction, while de-emphasizing achievement, power, and security relative to humans. Successful discriminant analysis differentiated the 4 LLMs' distinct value profiles. Further examination found the biased value profiles strongly predicted the LLMs' responses when presented with mental health dilemmas requiring choosing between opposing values. This provided further validation for the models embedding distinct motivational value-like constructs that shape their decision-making. Conclusions: This study leveraged the STBV to map the motivational value-like infrastructure underpinning leading LLMs. Although the study demonstrated the STBV can effectively characterize value-like infrastructure within LLMs, substantial divergence from human values raises ethical concerns about aligning these models with mental health applications. The biases toward certain cultural value sets pose risks if integrated without proper safeguards. For example, prioritizing universalism could promote unconditional acceptance even when clinically unwise. Furthermore, the differences between the LLMs underscore the need to standardize alignment processes to capture true cultural diversity. Thus, any responsible integration of LLMs into mental health care must account for their embedded biases and motivation mismatches to ensure equitable delivery across diverse populations. Achieving this will require transparency and refinement of alignment techniques to instill comprehensive human values.

Extending the Cross-Cultural Validity of the Theory of Basic Human Values with a Different Method of Measurement

Measuring the Four Higher-Order Values in Schwartz's Theory: Validation of a 17-Item Inventory

Developing and Evaluating the Social Axioms Survey in Eleven Countries

Values and Behavior Among Minorities in Southwest China: A Cross-Cultural Validation of the Refined Value Theory

Measurement Invariance of the Moral Vitalism Scale Across 28 Cultural Groups.

Investigating the Circumplex Structure of the Portrait Values Questionnaire (PVQ)

Values, Attitudes Toward Interpersonal Violence, and Interpersonal Violent Behavior

The Diversity of Value Construal: A Constructivist Approach to the Schwartz Theory of Basic Values

Cognitive Ability and Personal Values: A Large Sample Study of Schwartz's Values, HEXACO Personality, Age, and Gender

Values and Behavior: Strength and Structure of Relations

Relationships between personal human values and social value orientation

How and when do personal values guide our attitudes and sociality? Explaining cross-cultural variability in attitude-value linkages

Are Individual-Level and Country-Level Value Structures Different? Testing Hofstede’s Legacy With the Schwartz Value Survey

Validity Problems Comparing Values Across Cultures and Possible Solutions.

The Contribution of Peer Values to Children’s Values and Behavior

Investigating Human Values in Online Communities

Assessing the Alignment of Large Language Models With Human Values for Mental Health Integration: Cross-Sectional Study Using Schwartz's Theory of Basic Values

Developing Practical Procedures for the Measurement of Personal Values in Cross-Cultural Marketing

Cross-Cultural Differences and Similarities in Human Value Instantiation

Measuring Cultural Dimensions: External Validity and Internal Consistency of Hofstede's VSM 2013 Scales

Measuring sociocultural characteristics according to the six indicators of the Hofstede model: testing the toolkit for calculating values at the individual level