Simulating Policy Impacts: Developing a Generative Scenario Writing Method to Evaluate the Perceived Effects of Regulation

Julia Barnett,Kimon Kieslich,Nicholas Diakopoulos

2024-07-27

Abstract:The rapid advancement of AI technologies yields numerous future impacts on individuals and society. Policymakers are tasked to react quickly and establish policies that mitigate those impacts. However, anticipating the effectiveness of policies is a difficult task, as some impacts might only be observable in the future and respective policies might not be applicable to the future development of AI. In this work we develop a method for using large language models (LLMs) to evaluate the efficacy of a given piece of policy at mitigating specified negative impacts. We do so by using GPT-4 to generate scenarios both pre- and post-introduction of policy and translating these vivid stories into metrics based on human perceptions of impacts. We leverage an already established taxonomy of impacts of generative AI in the media environment to generate a set of scenario pairs both mitigated and non-mitigated by the transparency policy in Article 50 of the EU AI Act. We then run a user study (n=234) to evaluate these scenarios across four risk-assessment dimensions: severity, plausibility, magnitude, and specificity to vulnerable populations. We find that this transparency legislation is perceived to be effective at mitigating harms in areas such as labor and well-being, but largely ineffective in areas such as social cohesion and security. Through this case study we demonstrate the efficacy of our method as a tool to iterate on the effectiveness of policy for mitigating various negative impacts. We expect this method to be useful to researchers or other stakeholders who want to brainstorm the potential utility of different pieces of policy or other mitigation strategies.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the following issues:

1. **Develop a generative scenario writing method**: By leveraging large language models (such as GPT-4) to assess the effectiveness of specific policies in mitigating the negative impacts of AI technology. Specifically, this method generates scenario stories before and after policy implementation and translates these stories into human perception-based metrics.

2. **Evaluate the effectiveness of transparency policies**: The study particularly focuses on the transparency policies in Article 50 of the EU AI Act, assessing their effectiveness in various domains such as labor, welfare, social cohesion, and safety through user research.

3. **Provide decision support tools**: Offer researchers and other stakeholders a low-cost method to explore the potential utility of different policies or mitigation strategies, thereby helping them make better decisions before engaging in more costly evaluation methods (such as experiments or pilot policy deployments).

Through this method, the paper demonstrates how to use large language models to generate and rewrite scenario stories, thereby assessing the actual effects of policies in different impact areas. This provides policymakers with a new tool to better understand and evaluate the potential social impacts of AI technology at an early stage.