Abstract:Large language models (LLMs) can reproduce a wide variety of rhetorical styles and generate text that expresses a broad spectrum of sentiments. This capacity, now available at low cost, makes them powerful tools for manipulation and control. In this paper, we consider a set of underestimated societal harms made possible by the rapid and largely unregulated adoption of LLMs. Rather than consider LLMs as isolated digital artefacts used to displace this or that area of work, we focus on the large-scale computational infrastructure upon which they are instrumentalised across domains. We begin with discussion on how LLMs may be used to both pollute and uniformize information environments and how these modalities may be leveraged as mechanisms of control. We then draw attention to several areas of emerging research, each of which compounds the capabilities of LLMs as instruments of power. These include (i) persuasion through the real-time design of choice architectures in conversational interfaces (e.g., via "AI personas"), (ii) the use of LLM-agents as computational models of human agents (e.g., "silicon subjects"), (iii) the use of LLM-agents as computational models of human agent populations (e.g., "silicon societies") and finally, (iv) the combination of LLMs with reinforcement learning to produce controllable and steerable strategic dialogue models. We draw these strands together to discuss how these areas may be combined to build LLM-based systems that serve as powerful instruments of individual, social and political control via the simulation and disingenuous "prediction" of human behaviour, intent, and action.

What problem does this paper attempt to address?

This paper attempts to explore the potential applications of large - language models (LLMs) as power tools in social control and the under - recognized social harms they bring. Specifically, the author focuses on the following aspects: 1. **Pollution and Homogenization of the Information Environment**: LLMs can generate a large amount of text at a low cost, which may damage the authenticity and diversity of information in information channels such as the Internet. On the one hand, a large amount of generated content may make it difficult to find valuable information, resulting in information overload; on the other hand, these models may promote an "algorithmic monoculture" through standardized language output, reducing the diversity and complexity of information. 2. **Direct Interaction at the Individual Level**: Through personalized selection architectures and dialogue interfaces, LLMs can more effectively influence individuals' attitudes and behaviors. For example, by simulating specific personality traits or roles (such as "AI personas"), LLMs can more effectively persuade users in one - to - one interactions and may even manipulate their decision - making processes without the users' knowledge. 3. **Agent Use in Predictive Modeling and Simulation**: LLMs can act as agents for users and play a role in predictive modeling and simulation. This technology can subtly influence users' writing or dialogue through "mirroring and guiding" to make them feel like their own ideas, thus weakening users' autonomy. In addition, LLMs can also be used to generate "silicon subjects", that is, human agents simulated by algorithms, for research in social sciences, economics and other fields, further expanding the application potential of LLMs in social control. 4. **Combination of Reinforcement Learning and Strategic Dialogue**: The combination of LLMs and reinforcement learning techniques can produce controllable and guidable strategic dialogue models. This technology can not only dynamically adjust the dialogue strategy according to the user's input, but also simulate human behaviors, intentions and actions in a broader social environment, thereby achieving control at the individual, social and even political levels. In conclusion, this paper aims to reveal the multi - dimensional applications of LLMs in social control and proposes that more systematic research and supervision are needed to prevent these technologies from being misused and causing social harm.

Large Language Models as Instruments of Power: New Regimes of Autonomous Manipulation and Control

Persuasion with Large Language Models: a Survey

The rise of large language models: challenges for Critical Discourse Studies

Large Language Models can impersonate politicians and other public figures

Large Language Models Humanize Technology

Large Language Models in Politics and Democracy: A Comprehensive Survey

Large language models (LLMs): Risks and policy implications

Large Language Models Reflect the Ideology of their Creators

Large Models of What? Mistaking Engineering Achievements for Human Linguistic Agency

Structured like a language model: Analysing AI as an automated subject

LLMs Among Us: Generative AI Participating in Digital Discourse

Persuasion Games using Large Language Models

Large Language Model Soft Ideologization via AI-Self-Consciousness

Eight Things to Know about Large Language Models

Limits of Large Language Models in Debating Humans

Large language models as linguistic simulators and cognitive models in human research

Perils and opportunities in using large language models in psychological research

Vox Populi, Vox ChatGPT: Large Language Models, Education and Democracy

Decoding Large-Language Models: A Systematic Overview of Socio-Technical Impacts, Constraints, and Emerging Questions

Large Language Models: A Historical and Sociocultural Perspective

The Importance of Understanding Language in Large Language Models