This paper presents an argument that certain AI safety measures, rather than
mitigating existential risk, may instead exacerbate it. Under certain key
assumptions - the inevitability of AI failure, the expected correlation between
an AI system's power at the point of failure and the severity of the resulting
harm, and the tendency of safety measures to enable AI systems to become more
powerful before failing - safety efforts have negative expected utility. The
paper examines three response strategies: Optimism, Mitigation, and Holism.
Each faces challenges stemming from intrinsic features of the AI safety
landscape that we term Bottlenecking, the Perfection Barrier, and Equilibrium
Fluctuation. The surprising robustness of the argument forces a re-examination
of core assumptions around AI safety and points to several avenues for further
research.

AI 安全措施可能加剧而非减轻存在风险，对 AI 失败的不可避免性、失败点 AI 系统能力与伤害严重程度的预期相关性以及安全措施在失败前使 AI 系统更强大的倾向等核心假设提出负面预期效用。本文探讨了乐观主义、缓解和整体性三种应对策略，每种策略面临 AI 安全景观内固有特征所带来的挑战，例如瓶颈、完美障碍和平衡波动。该论点的意外稳健性迫使重新审视 AI 安全的核心假设，并指出了一些值得进一步研究的方向。

人工智能安全：通往末日的攀登？

AI Safety: A Climb To Armageddon?

Artificial general intelligence (AGI) does not yet exist, but given the pace
of technological development in artificial intelligence, it is projected to
reach human-level intelligence within roughly the next two decades. After that,
many experts expect it to far surpass human intelligence and to do so rapidly.
The prospect of superintelligent AGI poses an existential risk to humans
because there is no reliable method for ensuring that AGI goals stay aligned
with human goals. Drawing on publicly available forecaster and opinion data,
the author examines how experts and non-experts perceive risk from AGI. The
findings indicate that the perceived risk of a world catastrophe or extinction
from AGI is greater than for other existential risks. The increase in perceived
risk over the last year is also steeper for AGI than for other existential
threats (e.g., nuclear war or human-caused climate change). That AGI is a
pressing existential risk is something on which experts and non-experts agree,
but the basis for such agreement currently remains obscure.

人工通用智能（AGI）尚不存在，但鉴于人工智能技术发展的速度，预计在未来大约二十年内将达到人类水平的智能水平。之后，许多专家预计它将远远超过人类智能，并且迅速超过。超级智能 AGI 的前景对人类构成了存亡风险，因为目前没有可靠的方法来确保 AGI 的目标与人类目标保持一致。作者借鉴公开可用的预测者和观点数据，考察专家和非专家对 AGI 风险的感知。研究结果表明，与其他存在风险相比，人们对 AGI 造成的世界灾难或灭绝的感知风险更大。与其他存在威胁相比，AGI 的感知风险在过去一年中增加的速度也更快（例如核战争或人为引起的气候变化）。专家和非专家一致认为 AGI 是一种紧迫的存亡风险，但目前对于这种共识的基础仍然不清楚。

人工通用智能、存在风险和人类风险认知

Artificial General Intelligence, Existential Risk, and Human Risk  Perception

This report examines what I see as the core argument for concern about
existential risk from misaligned artificial intelligence. I proceed in two
stages. First, I lay out a backdrop picture that informs such concern. On this
picture, intelligent agency is an extremely powerful force, and creating agents
much more intelligent than us is playing with fire -- especially given that if
their objectives are problematic, such agents would plausibly have instrumental
incentives to seek power over humans. Second, I formulate and evaluate a more
specific six-premise argument that creating agents of this kind will lead to
existential catastrophe by 2070. On this argument, by 2070: (1) it will become
possible and financially feasible to build relevantly powerful and agentic AI
systems; (2) there will be strong incentives to do so; (3) it will be much
harder to build aligned (and relevantly powerful/agentic) AI systems than to
build misaligned (and relevantly powerful/agentic) AI systems that are still
superficially attractive to deploy; (4) some such misaligned systems will seek
power over humans in high-impact ways; (5) this problem will scale to the full
disempowerment of humanity; and (6) such disempowerment will constitute an
existential catastrophe. I assign rough subjective credences to the premises in
this argument, and I end up with an overall estimate of ~5% that an existential
catastrophe of this kind will occur by 2070. (May 2022 update: since making
this report public in April 2021, my estimate here has gone up, and is now at
>10%.)

本文探讨了关于人工智能失控所带来的灾难性威胁的核心论点，论述了超级智能对人类的潜在生存危险，提出了深度学习中存在问题的因素，并预测到在 2070 年前此问题的发展将导致人类的终极灭亡。