Generative, multimodal artificial intelligence (GenAI) offers transformative
potential across industries, but its misuse poses significant risks. Prior
research has shed light on the potential of advanced AI systems to be exploited
for malicious purposes. However, we still lack a concrete understanding of how
GenAI models are specifically exploited or abused in practice, including the
tactics employed to inflict harm. In this paper, we present a taxonomy of GenAI
misuse tactics, informed by existing academic literature and a qualitative
analysis of approximately 200 observed incidents of misuse reported between
January 2023 and March 2024. Through this analysis, we illuminate key and novel
patterns in misuse during this time period, including potential motivations,
strategies, and how attackers leverage and abuse system capabilities across
modalities (e.g. image, text, audio, video) in the wild.

通过对现有学术文献和大约 200 起滥用报告事件的定性分析，我们提出了一种基于深度学习和多模态人工智能的滥用策略分类方法，并揭示了滥用的关键模式、潜在动机和攻击者如何利用系统能力的方式。

生成人工智能滥用方法分类及来自实际数据的洞见

Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World  Data

Most task-oriented dialogue (TOD) benchmarks assume users that know exactly
how to use the system by constraining the user behaviors within the system's
capabilities via strict user goals, namely "user familiarity" bias. This data
bias deepens when it combines with data-driven TOD systems, as it is impossible
to fathom the effect of it with existing static evaluations. Hence, we conduct
an interactive user study to unveil how vulnerable TOD systems are against
realistic scenarios. In particular, we compare users with 1) detailed goal
instructions that conform to the system boundaries (closed-goal) and 2) vague
goal instructions that are often unsupported but realistic (open-goal). Our
study reveals that conversations in open-goal settings lead to catastrophic
failures of the system, in which 92% of the dialogues had significant issues.
Moreover, we conduct a thorough analysis to identify distinctive features
between the two settings through error annotation. From this, we discover a
novel "pretending" behavior, in which the system pretends to handle the user
requests even though they are beyond the system's capabilities. We discuss its
characteristics and toxicity while emphasizing transparency and a fallback
strategy for robust TOD systems.

通过交互用户研究，比较了符合系统边界的具体目标说明和不支持但更真实的目标说明两种用户情境下，任务导向型对话系统的漏洞和错误，发现了一种新的机器行为 —— 伪装，在强调透明度和回退策略的同时呼吁建立稳健的任务导向型对话系统。