The Voice: Lessons on Trustworthy Conversational Agents from "Dune"

Philip Feldman

DOI: https://doi.org/10.1145/3640794.3665890

2024-07-10

Abstract:The potential for untrustworthy conversational agents presents a significant threat for covert social manipulation. Taking inspiration from Frank Herbert's "Dune", where the Bene Gesserit Sisterhood uses the Voice for influence, manipulation, and control of people, we explore how generative AI provides a way to implement individualized influence at industrial scales. Already, these models can manipulate communication across text, image, speech, and most recently video. They are rapidly becoming affordable enough for any organization of even moderate means to train and deploy. If employed by malicious actors, they risk becoming powerful tools for shaping public opinion, sowing discord, and undermining organizations from companies to governments. As researchers and developers, it is crucial to recognize the potential for such weaponization and to explore strategies for prevention, detection, and defense against these emerging forms of sociotechnical manipulation.

Computers and Society,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to deal with the potential threats of generative AI in terms of social manipulation and information control**. Specifically, the author draws inspiration from the "Voice" technology of the Bene Gesserit sisterhood in the science - fiction novel "Dune" to explore how generative AI can carry out personalized influence on an industrial scale and may be used by malicious actors to manipulate public opinion, create divisions and undermine organizations (from companies to governments).

### Main problems:
1. **Potential threats of generative AI**:
- Generative AI models (such as large - language models, deepfakes, etc.) are already able to conduct extensive social manipulation through text, image, voice and even video.
- The costs of these models are gradually decreasing, enabling even organizations with limited resources to train and deploy these models.
- If exploited by malicious actors, these tools may become powerful weapons for shaping public opinion, inciting discord and weakening various organizations.

2. **Long - term social and technological manipulation risks**:
- The paper points out that generative AI can, like the Bene Gesserit sisterhood in "Dune", influence individuals and society in a subtle way in the long term, gradually changing people's attitudes and beliefs without arousing suspicion.
- Such long - term manipulation may have a profound impact on social structures, political systems and cultures, similar to the millennium plan described in "Dune".

3. **Defense and detection strategies**:
- The author emphasizes that researchers and developers must recognize this weaponization possibility and explore methods to prevent, detect and defend against these emerging forms of socio - technological manipulation.
- Specifically, the paper proposes the concept of "white - hat AI", that is, to develop AI systems that can identify and resist manipulative content, similar to the "Truthsayer" in "Dune", to protect the public from malicious AI.

### Conclusion:
The core of the paper lies in calling on the academic and industrial communities to pay attention to the potential abuse risks of generative AI and proposing the need to develop corresponding defense measures to prevent these technologies from being used for malicious purposes. The author believes that understanding the working principles of "black - hat AI" is crucial for designing effective "white - hat AI", thus ensuring the safety and credibility of AI technology.

### Key citations:
- "As researchers and developers, we must recognize this weaponization potential and explore strategies to prevent, detect and defend against these emerging forms of socio - technological manipulation."
- "Generative AI models possess patience, influence and subtlety and can approach Bene Gesserit's technology, especially in spreading and manipulating narratives."

In this way, the paper not only explores the potential threats of generative AI but also proposes specific countermeasures, aiming to provide guidance for future research and practice.