Foundation models are powerful technologies: how they are released publicly
directly shapes their societal impact. In this position paper, we focus on open
foundation models, defined here as those with broadly available model weights
(e.g. Llama 2, Stable Diffusion XL). We identify five distinctive properties
(e.g. greater customizability, poor monitoring) of open foundation models that
lead to both their benefits and risks. Open foundation models present
significant benefits, with some caveats, that span innovation, competition, the
distribution of decision-making power, and transparency. To understand their
risks of misuse, we design a risk assessment framework for analyzing their
marginal risk. Across several misuse vectors (e.g. cyberattacks, bioweapons),
we find that current research is insufficient to effectively characterize the
marginal risk of open foundation models relative to pre-existing technologies.
The framework helps explain why the marginal risk is low in some cases,
clarifies disagreements about misuse risks by revealing that past work has
focused on different subsets of the framework with different assumptions, and
articulates a way forward for more constructive debate. Overall, our work helps
support a more grounded assessment of the societal impact of open foundation
models by outlining what research is needed to empirically validate their
theoretical benefits and risks.

开放基础模型具有创新、竞争、决策权分配和透明度等显著优势，但当前的研究还不足以有效地表征相对于现有技术的边际风险，因此需要进一步的研究来实证验证其理论上的利益和风险。

开放基础模型的社会影响

On the Societal Impact of Open Foundation Models

The expanding application of Artificial Intelligence (AI) in scientific
fields presents unprecedented opportunities for discovery and innovation.
However, this growth is not without risks. AI models in science, if misused,
can amplify risks like creation of harmful substances, or circumvention of
established regulations. In this study, we aim to raise awareness of the
dangers of AI misuse in science, and call for responsible AI development and
use in this domain. We first itemize the risks posed by AI in scientific
contexts, then demonstrate the risks by highlighting real-world examples of
misuse in chemical science. These instances underscore the need for effective
risk management strategies. In response, we propose a system called SciGuard to
control misuse risks for AI models in science. We also propose a red-teaming
benchmark SciMT-Safety to assess the safety of different systems. Our proposed
SciGuard shows the least harmful impact in the assessment without compromising
performance in benign tests. Finally, we highlight the need for a
multidisciplinary and collaborative effort to ensure the safe and ethical use
of AI models in science. We hope that our study can spark productive
discussions on using AI ethically in science among researchers, practitioners,
policymakers, and the public, to maximize benefits and minimize the risks of
misuse.

该研究旨在提高科学领域中人工智能滥用的危险，并呼吁在该领域中负责任地开发和使用人工智能。通过列举科学环境中人工智能带来的风险，并通过化学科学领域的实际滥用示例来展示这些风险，高亮显示了有效的风险管理策略的需求。为此，我们提出了一个名为 SciGuard 的系统来控制科学中人工智能模型的滥用风险，并提出了一个名为 SciMT-Safety 的红队基准来评估不同系统的安全性。我们的提议在评估中显示出最小的有害影响，并且在良性测试中不会影响性能。最后，我们强调了跨学科和合作努力来确保在科学中安全和道德地使用人工智能模型的需求。我们希望我们的研究能够激发研究人员、从业者、决策者和公众之间就科学中的人工智能的道德使用展开有建设性的讨论，以最大化利益，最小化滥用风险。

控制科学中人工智能潜在滥用的风险

Control Risk for Potential Misuse of Artificial Intelligence in Science

Despite remarkable success in various applications, large language models
(LLMs) are vulnerable to adversarial jailbreaks that make the safety guardrails
void. However, previous studies for jailbreaks usually resort to brute-force
optimization or extrapolations of a high computation cost, which might not be
practical or effective. In this paper, inspired by the Milgram experiment that
individuals can harm another person if they are told to do so by an
authoritative figure, we disclose a lightweight method, termed as
DeepInception, which can easily hypnotize LLM to be a jailbreaker and unlock
its misusing risks. Specifically, DeepInception leverages the personification
ability of LLM to construct a novel nested scene to behave, which realizes an
adaptive way to escape the usage control in a normal scenario and provides the
possibility for further direct jailbreaks. Empirically, we conduct
comprehensive experiments to show its efficacy. Our DeepInception can achieve
competitive jailbreak success rates with previous counterparts and realize a
continuous jailbreak in subsequent interactions, which reveals the critical
weakness of self-losing on both open/closed-source LLMs like Falcon, Vicuna,
Llama-2, and GPT-3.5/4/4V. Our investigation appeals that people should pay
more attention to the safety aspects of LLMs and a stronger defense against
their misuse risks. The code is publicly available at:
this https URL

LLMs 容易受到破解攻击，本研究提出了一种 DeepInception 方法来解除 LLMs 在使用控制方面的限制，揭示了它们的关键弱点，需加强安全性的防御。