Skip connections made the training of very deep networks possible and have
become an indispensable component in a variety of neural architectures. A
completely satisfactory explanation for their success remains elusive. Here, we
present a novel explanation for the benefits of skip connections in training
very deep networks. The difficulty of training deep networks is partly due to
the singularities caused by the non-identifiability of the model. Several such
singularities have been identified in previous works: (i) overlap singularities
caused by the permutation symmetry of nodes in a given layer, (ii) elimination
singularities corresponding to the elimination, i.e. consistent deactivation,
of nodes, (iii) singularities generated by the linear dependence of the nodes.
These singularities cause degenerate manifolds in the loss landscape that slow
down learning. We argue that skip connections eliminate these singularities by
breaking the permutation symmetry of nodes, by reducing the possibility of node
elimination and by making the nodes less linearly dependent. Moreover, for
typical initializations, skip connections move the network away from the
"ghosts" of these singularities and sculpt the landscape around them to
alleviate the learning slow-down. These hypotheses are supported by evidence
from simplified models, as well as from experiments with deep networks trained
on real-world datasets.

研究探讨了 skip connection 对深层网络训练中的奇异点造成的影响，证明了 skip connection 可以消除这些奇异点，并在实验中得到了验证。

跳跃连接消除奇异点

Skip Connections Eliminate Singularities

In this paper, we empirically explore the effects of various kinds of skip
connections in stacked bidirectional LSTMs for sequential tagging. We
investigate three kinds of skip connections connecting to LSTM cells: (a) skip
connections to the gates, (b) skip connections to the internal states and (c)
skip connections to the cell outputs. We present comprehensive experiments
showing that skip connections to cell outputs outperform the remaining two.
Furthermore, we observe that using gated identity functions as skip mappings
works pretty well. Based on this novel skip connections, we successfully train
deep stacked bidirectional LSTM models and obtain state-of-the-art results on
CCG supertagging and comparable results on POS tagging.

本文探讨了在双向 LSTMs 中使用不同类型的跳跃连接对于序列标注的影响，并通过全面实验发现跳跃连接到细胞输出优于其他两种方法。此外，本文还注意到使用带门控的恒等函数作为跳跃映射可以获得很好的效果。基于这些新颖的跳跃连接，我们成功地训练了深度堆叠的双向 LSTM 模型，并在 CCG 超级标记上获得了最新的成果，在 POS 标记上获得了可比较的结果。

序列标注中跳连结的实证探索

An Empirical Exploration of Skip Connections for Sequential Tagging

In this paper, we study the influence of both long and short skip connections
on Fully Convolutional Networks (FCN) for biomedical image segmentation. In
standard FCNs, only long skip connections are used to skip features from the
contracting path to the expanding path in order to recover spatial information
lost during downsampling. We extend FCNs by adding short skip connections, that
are similar to the ones introduced in residual networks, in order to build very
deep FCNs (of hundreds of layers). A review of the gradient flow confirms that
for a very deep FCN it is beneficial to have both long and short skip
connections. Finally, we show that a very deep FCN can achieve
near-to-state-of-the-art results on the EM dataset without any further
post-processing.

本研究探讨长短跳跃连接对全卷积网络进行生物医学图像分割的影响，发现在非常深的 FCN 中同时使用长、短跳跃连接是有益的，并展示了非常深的 FCN 在 EM 数据集上实现了接近最新成果的结果而无需进一步的后处理。

生物医学图像分割中跳跃连接的重要性

The Importance of Skip Connections in Biomedical Image Segmentation

Very deep convolutional neural networks introduced new problems like
vanishing gradient and degradation. The recent successful contributions towards
solving these problems are Residual and Highway Networks. These networks
introduce skip connections that allow the information (from the input or those
learned in earlier layers) to flow more into the deeper layers. These very deep
models have lead to a considerable decrease in test errors, on benchmarks like
ImageNet and COCO. In this paper, we propose the use of exponential linear unit
instead of the combination of ReLU and Batch Normalization in Residual
Networks. We show that this not only speeds up learning in Residual Networks
but also improves the accuracy as the depth increases. It improves the test
error on almost all data sets, like CIFAR-10 and CIFAR-100

本文旨在指出在深度卷积神经网络中引入跳过连接从而解决挥发的渐变问题的新方案，并提出将指数线性单元用于残差网络，以显著减小测试误差。