We study the effect of width on the dynamics of feature-learning neural
networks across a variety of architectures and datasets. Early in training,
wide neural networks trained on online data have not only identical loss curves
but also agree in their point-wise test predictions throughout training. For
simple tasks such as CIFAR-5m this holds throughout training for networks of
realistic widths. We also show that structural properties of the models,
including internal representations, preactivation distributions, edge of
stability phenomena, and large learning rate effects are consistent across
large widths. This motivates the hypothesis that phenomena seen in realistic
models can be captured by infinite-width, feature-learning limits. For harder
tasks (such as ImageNet and language modeling), and later training times,
finite-width deviations grow systematically. Two distinct effects cause these
deviations across widths. First, the network output has
initialization-dependent variance scaling inversely with width, which can be
removed by ensembling networks. We observe, however, that ensembles of narrower
networks perform worse than a single wide network. We call this the bias of
narrower width. We conclude with a spectral perspective on the origin of this
finite-width bias.

研究神经网络的宽度对特征学习动态的影响，探究网络结构，内部表示，预激活分布，稳定现象的一致性，以及相应的有限宽度偏差和频谱视角。

特征学习网络在现实规模下的宽度一致性

Feature-Learning Networks Are Consistent Across Widths At Realistic  Scales

Recent years have seen a dramatic expansion of tasks and datasets posed as
question answering, from reading comprehension, semantic role labeling, and
even machine translation, to image and video understanding. With this
expansion, there are many differing views on the utility and definition of
"question answering" itself. Some argue that its scope should be narrow, or
broad, or that it is overused in datasets today. In this opinion piece, we
argue that question answering should be considered a format which is sometimes
useful for studying particular phenomena, not a phenomenon or task in itself.
We discuss when a task is correctly described as question answering, and when a
task is usefully posed as question answering, instead of using some other
format.

本文探讨了任务和数据集的扩展与多样性，讨论了问题回答作为一种用于研究特定现象的格式，而非独立于其自身的一种任务或现象，同时阐述了何时正确地将任务描述为问题回答，何时有必要将任务表述为问题回答而不是其他格式。

问答是一种格式；何时有用？

Question Answering is a Format; When is it Useful?

Networks have been used to model many real-world phenomena to better
understand the phenomena and to guide experiments in order to predict their
behavior. Since incorrect models lead to incorrect predictions, it is vital to
have a correct model. As a result, new techniques and models for analyzing and
modeling real-world networks have recently been introduced. One example of
large and complex networks involves protein-protein interaction (PPI) networks.
We demonstrate that the currently popular scale-free model of PPI networks
fails to fit the data in several respects. We show that a random geometric
model provides a much more accurate model of the PPI data.

以蛋白质相互作用网络为例，网络模型的正确性对于预测其行为至关重要。然而，研究表明目前广受欢迎的自组织网络模型不能很好拟合 PPI 数据，相对而言，随机几何模型更准确。