Abstract:The recent development of large language models (LLMs) has spurred discussions about whether LLM-generated "synthetic samples" could complement or replace traditional surveys, considering their training data potentially reflects attitudes and behaviors prevalent in the population. A number of mostly US-based studies have prompted LLMs to mimic survey respondents, with some of them finding that the responses closely match the survey data. However, several contextual factors related to the relationship between the respective target population and LLM training data might affect the generalizability of such findings. In this study, we investigate the extent to which LLMs can estimate public opinion in Germany, using the example of vote choice. We generate a synthetic sample of personas matching the individual characteristics of the 2017 German Longitudinal Election Study respondents. We ask the LLM GPT-3.5 to predict each respondent's vote choice and compare these predictions to the survey-based estimates on the aggregate and subgroup levels. We find that GPT-3.5 does not predict citizens' vote choice accurately, exhibiting a bias towards the Green and Left parties. While the LLM captures the tendencies of "typical" voter subgroups, such as partisans, it misses the multifaceted factors swaying individual voter choices. By examining the LLM-based prediction of voting behavior in a new context, our study contributes to the growing body of research about the conditions under which LLMs can be leveraged for studying public opinion. The findings point to disparities in opinion representation in LLMs and underscore the limitations in applying them for public opinion estimation.

Large Language Models (LLMs) as Agents for Augmented Democracy

Large Language Models Can Be Used to Estimate the Latent Positions of Politicians

Large Language Models in Politics and Democracy: A Comprehensive Survey

LLM Voting: Human Choices and AI Collective Decision Making

A Large-scale Empirical Study on Large Language Models for Election Prediction

Large Language Models' Detection of Political Orientation in Newspapers

Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study

Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks

Large Language Models as Subpopulation Representative Models: A Review

Donald Trumps in the Virtual Polls: Simulating and Predicting Public Opinions in Surveys Using Large Language Models

Can LLMs Help Predict Elections? (Counter)Evidence from the World's Largest Democracy

Vox Populi, Vox AI? Using Language Models to Estimate German Public Opinion

AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction

United in Diversity? Contextual Biases in LLM-Based Predictions of the 2024 European Parliament Elections

The Political Preferences of LLMs

Positioning Political Texts with Large Language Models by Asking and Averaging

Assessing Political Bias in Large Language Models

Can Large Language Models Transform Computational Social Science?

Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs

Can LLMs advance democratic values?

Large Language Models: A New Approach for Privacy Policy Analysis at Scale