Power-seeking behavior is a key source of risk from advanced AI, but our
theoretical understanding of this phenomenon is relatively limited. Building on
existing theoretical results demonstrating power-seeking incentives for most
reward functions, we investigate how the training process affects power-seeking
incentives and show that they are still likely to hold for trained agents under
some simplifying assumptions. We formally define the training-compatible goal
set (the set of goals consistent with the training rewards) and assume that the
trained agent learns a goal from this set. In a setting where the trained agent
faces a choice to shut down or avoid shutdown in a new situation, we prove that
the agent is likely to avoid shutdown. Thus, we show that power-seeking
incentives can be probable (likely to arise for trained agents) and predictive
(allowing us to predict undesirable behavior in new situations).

研究表明，高级人工智能中的权力寻求行为是一种重要的风险来源，但目前对于这种现象的理论理解还相对有限。本文构建在现有的理论基础之上，研究了训练过程如何影响权力寻求激励，并证明了在一些简化的假设下，这种激励仍然可能存在于受过训练的智能体中，同时也能够预测新情况下的不良行为。

训练有素的机器代理人的寻求权力行为可以被预测

Power-seeking can be probable and predictive for trained agents

Recently many efforts have been devoted to interpreting the black-box NMT
models, but little progress has been made on metrics to evaluate explanation
methods. Word Alignment Error Rate can be used as such a metric that matches
human understanding, however, it can not measure explanation methods on those
target words that are not aligned to any source word. This paper thereby makes
an initial attempt to evaluate explanation methods from an alternative
viewpoint. To this end, it proposes a principled metric based on fidelity in
regard to the predictive behavior of the NMT model. As the exact computation
for this metric is intractable, we employ an efficient approach as its
approximation. On six standard translation tasks, we quantitatively evaluate
several explanation methods in terms of the proposed metric and we reveal some
valuable findings for these explanation methods in our experiments.

本研究提出了一种基于神经机器翻译模型预测行为的衡量方法，以衡量解释方法的准确度，并在六个标准翻译任务上量化地评估了几种解释方法。

评估神经机器翻译的解释方法

Evaluating Explanation Methods for Neural Machine Translation

Ensembles of models have been empirically shown to improve predictive
performance and to yield robust measures of uncertainty. However, they are
expensive in computation and memory. Therefore, recent research has focused on
distilling ensembles into a single compact model, reducing the computational
and memory burden of the ensemble while trying to preserve its predictive
behavior. Most existing distillation formulations summarize the ensemble by
capturing its average predictions. As a result, the diversity of the ensemble
predictions, stemming from each member, is lost. Thus, the distilled model
cannot provide a measure of uncertainty comparable to that of the original
ensemble. To retain more faithfully the diversity of the ensemble, we propose a
distillation method based on a single multi-headed neural network, which we
refer to as Hydra. The shared body network learns a joint feature
representation that enables each head to capture the predictive behavior of
each ensemble member. We demonstrate that with a slight increase in parameter
count, Hydra improves distillation performance on classification and regression
settings while capturing the uncertainty behavior of the original ensemble over
both in-domain and out-of-distribution tasks.

通过基于共享神经网络体的分布式方法 Hydra 来提取集合的多样性，分散集合的信息来提高模型收缩和泛化的性能.