The expanding application of Artificial Intelligence (AI) in scientific
fields presents unprecedented opportunities for discovery and innovation.
However, this growth is not without risks. AI models in science, if misused,
can amplify risks like creation of harmful substances, or circumvention of
established regulations. In this study, we aim to raise awareness of the
dangers of AI misuse in science, and call for responsible AI development and
use in this domain. We first itemize the risks posed by AI in scientific
contexts, then demonstrate the risks by highlighting real-world examples of
misuse in chemical science. These instances underscore the need for effective
risk management strategies. In response, we propose a system called SciGuard to
control misuse risks for AI models in science. We also propose a red-teaming
benchmark SciMT-Safety to assess the safety of different systems. Our proposed
SciGuard shows the least harmful impact in the assessment without compromising
performance in benign tests. Finally, we highlight the need for a
multidisciplinary and collaborative effort to ensure the safe and ethical use
of AI models in science. We hope that our study can spark productive
discussions on using AI ethically in science among researchers, practitioners,
policymakers, and the public, to maximize benefits and minimize the risks of
misuse.

该研究旨在提高科学领域中人工智能滥用的危险，并呼吁在该领域中负责任地开发和使用人工智能。通过列举科学环境中人工智能带来的风险，并通过化学科学领域的实际滥用示例来展示这些风险，高亮显示了有效的风险管理策略的需求。为此，我们提出了一个名为 SciGuard 的系统来控制科学中人工智能模型的滥用风险，并提出了一个名为 SciMT-Safety 的红队基准来评估不同系统的安全性。我们的提议在评估中显示出最小的有害影响，并且在良性测试中不会影响性能。最后，我们强调了跨学科和合作努力来确保在科学中安全和道德地使用人工智能模型的需求。我们希望我们的研究能够激发研究人员、从业者、决策者和公众之间就科学中的人工智能的道德使用展开有建设性的讨论，以最大化利益，最小化滥用风险。