AI systems are becoming increasingly intertwined with human life. In order to
effectively collaborate with humans and ensure safety, AI systems need to be
able to understand, interpret and predict human moral judgments and decisions.
Human moral judgments are often guided by rules, but not always. A central
challenge for AI safety is capturing the flexibility of the human moral mind --
the ability to determine when a rule should be broken, especially in novel or
unusual situations. In this paper, we present a novel challenge set consisting
of rule-breaking question answering (RBQA) of cases that involve potentially
permissible rule-breaking -- inspired by recent moral psychology studies. Using
a state-of-the-art large language model (LLM) as a basis, we propose a novel
moral chain of thought (MORALCOT) prompting strategy that combines the
strengths of LLMs with theories of moral reasoning developed in cognitive
science to predict human moral judgments. MORALCOT outperforms seven existing
LLMs by 6.2% F1, suggesting that modeling human reasoning might be necessary to
capture the flexibility of the human moral mind. We also conduct a detailed
error analysis to suggest directions for future work to improve AI safety using
RBQA. Our data is open-sourced at
this https URL and code at
this https URL

为了能够有效地与人类协作并确保安全，人工智能系统需要能够理解、解释和预测人类的道德判断和决策。为了解决这一挑战，本文提出了一个基于最新的道德心理学研究的规则破坏问题回答 (RBQA) 挑战集，并使用最先进的大型语言模型 (LLMS) 作为基础，提出了一个新的 MORALCOT 策略以预测人类道德判断。