In an effort to inform the discussion surrounding existential risks from AI,
we formulate Extinction-level Goodhart's Law as "Virtually any goal
specification, pursued to the extreme, will result in the extinction of
humanity", and we aim to understand which formal models are suitable for
investigating this hypothesis. Note that we remain agnostic as to whether
Extinction-level Goodhart's Law holds or not. As our key contribution, we
identify a set of conditions that are necessary for a model that aims to be
informative for evaluating specific arguments for Extinction-level Goodhart's
Law. Since each of the conditions seems to significantly contribute to the
complexity of the resulting model, formally evaluating the hypothesis might be
exceedingly difficult. This raises the possibility that whether the risk of
extinction from artificial intelligence is real or not, the underlying dynamics
might be invisible to current scientific methods.

对于人工智能的存在风险进行讨论，提出了灭绝级古哈特定律的概念，即 “几乎任何目标规定，如果被极端追求，将导致人类的灭绝”，并旨在研究哪些形式模型适合探究这一假设。我们明确指出对于灭绝级古哈特定律是否成立我们持中立态度。作为我们的关键贡献，我们确定了一组用于评估灭绝级古哈特定律的具体论点的模型所必需的条件。由于这些条件似乎显著增加了结果模型的复杂性，因此正式评估该假设可能异常困难。这引发了这样一种可能性，即无论人工智能的灭绝风险是否真实存在，底层动态可能在当前的科学方法中是看不见的。