A Large Language Model~(LLM) is considered consistent if semantically
equivalent prompts produce semantically equivalent responses. Despite recent
advancements showcasing the impressive capabilities of LLMs in conversational
systems, we show that even state-of-the-art LLMs are highly inconsistent in
their generations, questioning their reliability. Prior research has tried to
measure this with task-specific accuracies. However, this approach is
unsuitable for moral scenarios, such as the trolley problem, with no
``correct'' answer. To address this issue, we propose a novel
information-theoretic measure called Semantic Graph Entropy~(SGE) to measure
the consistency of an LLM in moral scenarios. We leverage ``Rules of
Thumb''~(RoTs) to explain a model's decision-making strategies and further
enhance our metric. Compared to existing consistency metrics, SGE correlates
better with human judgments across five LLMs. In the future, we aim to
investigate the root causes of LLM inconsistencies and propose improvements.

大型语言模型在道德场景中呈现高度不一致性，因此提出了一种新的信息熵测量方法 —— 语义图熵（SGE）。通过使用模型的决策策略解释的 “经验法则”（RoTs），我们的度量与人类判断更好地相关，并计划研究和改进 LLM 不一致性的根本原因。