Intuitively, obedience -- following the order that a human gives -- seems
like a good property for a robot to have. But, we humans are not perfect and we
may give orders that are not best aligned to our preferences. We show that when
a human is not perfectly rational then a robot that tries to infer and act
according to the human's underlying preferences can always perform better than
a robot that simply follows the human's literal order. Thus, there is a
tradeoff between the obedience of a robot and the value it can attain for its
owner. We investigate how this tradeoff is impacted by the way the robot infers
the human's preferences, showing that some methods err more on the side of
obedience than others. We then analyze how performance degrades when the robot
has a misspecified model of the features that the human cares about or the
level of rationality of the human. Finally, we study how robots can start
detecting such model misspecification. Overall, our work suggests that there
might be a middle ground in which robots intelligently decide when to obey
human orders, but err on the side of obedience.

研究发现，当人不完全理性时，试图推断并根据人类的潜在偏好行动的机器人总是比简单遵循人类命令的机器人表现更好，因此，机器人的服从度与其所有者获得的价值之间存在权衡。