The rapid advancement of AI technologies yields numerous future impacts on
individuals and society. Policy-makers are therefore tasked to react quickly
and establish policies that mitigate those impacts. However, anticipating the
effectiveness of policies is a difficult task, as some impacts might only be
observable in the future and respective policies might not be applicable to the
future development of AI. In this work we develop a method for using large
language models (LLMs) to evaluate the efficacy of a given piece of policy at
mitigating specified negative impacts. We do so by using GPT-4 to generate
scenarios both pre- and post-introduction of policy and translating these vivid
stories into metrics based on human perceptions of impacts. We leverage an
already established taxonomy of impacts of generative AI in the media
environment to generate a set of scenario pairs both mitigated and
non-mitigated by the transparency legislation of Article 50 of the EU AI Act.
We then run a user study (n=234) to evaluate these scenarios across four
risk-assessment dimensions: severity, plausibility, magnitude, and specificity
to vulnerable populations. We find that this transparency legislation is
perceived to be effective at mitigating harms in areas such as labor and
well-being, but largely ineffective in areas such as social cohesion and
security. Through this case study on generative AI harms we demonstrate the
efficacy of our method as a tool to iterate on the effectiveness of policy on
mitigating various negative impacts. We expect this method to be useful to
researchers or other stakeholders who want to brainstorm the potential utility
of different pieces of policy or other mitigation strategies.

AI 技术、政策、语言模型、负面影响和透明度立法是本文的关键词，本文利用大型语言模型评估政策的有效性以减轻负面影响，并通过案例研究证明了方法的有效性。