Strong empirical evidence that one machine-learning algorithm A outperforms
another one B ideally calls for multiple trials optimizing the learning
pipeline over sources of variation such as data sampling, data augmentation,
parameter initialization, and hyperparameters choices. This is prohibitively
expensive, and corners are cut to reach conclusions. We model the whole
benchmarking process, revealing that variance due to data sampling, parameter
initialization and hyperparameter choice impact markedly the results. We
analyze the predominant comparison methods used today in the light of this
variance. We show a counter-intuitive result that adding more sources of
variation to an imperfect estimator approaches better the ideal estimator at a
51 times reduction in compute cost. Building on these results, we study the
error rate of detecting improvements, on five different deep-learning
tasks/architectures. This study leads us to propose recommendations for
performance comparisons.

通过模拟对比机器学习算法的整个基准测试过程，我们发现数据采样、参数初始化和超参数选择对结果的影响显著。进一步分析今天所用的主要比较方法，我们提出一种反直觉的结果，即在不增加计算成本的情况下，将更多的变化源添加到不完美的估计器中可以接近更理想的估计器。通过五个不同的深度学习任务和架构，分析改进检测错误率，提出了性能比较的建议。

机器学习基准测试中方差的考虑

Accounting for Variance in Machine Learning Benchmarks

Since Alan Turing envisioned Artificial Intelligence (AI) [1], a major
driving force behind technical progress has been competition with human
cognition. Historical milestones have been frequently associated with computers
matching or outperforming humans in difficult cognitive tasks (e.g. face
recognition [2], personality classification [3], driving cars [4], or playing
video games [5]), or defeating humans in strategic zero-sum encounters (e.g.
Chess [6], Checkers [7], Jeopardy! [8], Poker [9], or Go [10]). In contrast,
less attention has been given to developing autonomous machines that establish
mutually cooperative relationships with people who may not share the machine's
preferences. A main challenge has been that human cooperation does not require
sheer computational power, but rather relies on intuition [11], cultural norms
[12], emotions and signals [13, 14, 15, 16], and pre-evolved dispositions
toward cooperation [17], common-sense mechanisms that are difficult to encode
in machines for arbitrary contexts. Here, we combine a state-of-the-art
machine-learning algorithm with novel mechanisms for generating and acting on
signals to produce a new learning algorithm that cooperates with people and
other machines at levels that rival human cooperation in a variety of
two-player repeated stochastic games. This is the first general-purpose
algorithm that is capable, given a description of a previously unseen game
environment, of learning to cooperate with people within short timescales in
scenarios previously unanticipated by algorithm designers. This is achieved
without complex opponent modeling or higher-order theories of mind, thus
showing that flexible, fast, and general human-machine cooperation is
computationally achievable using a non-trivial, but ultimately simple, set of
algorithmic mechanisms.

通过创新的信号生成和操作机制，结合最先进的机器学习算法，实现了计算机在人机互动博弈中与人类合作的能力，无需复杂的对手建模或心灵理论。

与机器合作

Cooperating with Machines

Recommender systems improve access to relevant products and information by
making personalized suggestions based on previous examples of a user's likes
and dislikes. Most existing recommender systems use social filtering methods
that base recommendations on other users' preferences. By contrast,
content-based methods use information about an item itself to make suggestions.
This approach has the advantage of being able to recommended previously unrated
items to users with unique interests and to provide explanations for its
recommendations. We describe a content-based book recommending system that
utilizes information extraction and a machine-learning algorithm for text
categorization. Initial experimental results demonstrate that this approach can
produce accurate recommendations.

该研究描述了一种利用信息提取和机器学习算法进行文本分类的基于内容的书籍推荐系统，初步实验结果表明该方法可以产生准确的推荐。