Recent evidence reveals that Neural Machine Translation (NMT) models with
deeper neural networks can be more effective but are difficult to train. In
this paper, we present a MultiScale Collaborative (MSC) framework to ease the
training of NMT models that are substantially deeper than those used
previously. We explicitly boost the gradient back-propagation from top to
bottom levels by introducing a block-scale collaboration mechanism into deep
NMT models. Then, instead of forcing the whole encoder stack directly learns a
desired representation, we let each encoder block learns a fine-grained
representation and enhance it by encoding spatial dependencies using a
context-scale collaboration. We provide empirical evidence showing that the MSC
nets are easy to optimize and can obtain improvements of translation quality
from considerably increased depth. On IWSLT translation tasks with three
translation directions, our extremely deep models (with 72-layer encoders)
surpass strong baselines by +2.2~+3.1 BLEU points. In addition, our deep MSC
achieves a BLEU score of 30.56 on WMT14 English-German task that significantly
outperforms state-of-the-art deep NMT models.

本文提出了一种名为多尺度协作（MultiScale Collaborative）的框架，通过引入块尺度协作机制和上下文尺度协作增强梯度反向传播，并让每个编码器块学习细粒度的表示，以增加神经机器翻译模型的深度，从而提高翻译质量，并在多项任务中证明了该方法的有效性。

多尺度协同深度模型用于神经机器翻译

Multiscale Collaborative Deep Models for Neural Machine Translation

Although deep convolutional neural network has been proved to efficiently
eliminate coding artifacts caused by the coarse quantization of traditional
codec, it's difficult to train any neural network in front of the encoder for
gradient's back-propagation. In this paper, we propose an end-to-end image
compression framework based on convolutional neural network to resolve the
problem of non-differentiability of the quantization function in the standard
codec. First, the feature description neural network is used to get a valid
description in the low-dimension space with respect to the ground-truth image
so that the amount of image data is greatly reduced for storage or
transmission. After image's valid description, standard image codec such as
JPEG is leveraged to further compress image, which leads to image's great
distortion and compression artifacts, especially blocking artifacts, detail
missing, blurring, and ringing artifacts. Then, we use a post-processing neural
network to remove these artifacts. Due to the challenge of directly learning a
non-linear function for a standard codec based on convolutional neural network,
we propose to learn a virtual codec neural network to approximate the
projection from the valid description image to the post-processed compressed
image, so that the gradient could be efficiently back-propagated from the
post-processing neural network to the feature description neural network during
training. Meanwhile, an advanced learning algorithm is proposed to train our
deep neural networks for compression. Obviously, the priority of the proposed
method is compatible with standard existing codecs and our learning strategy
can be easily extended into these codecs based on convolutional neural network.
Experimental results have demonstrated the advances of the proposed method as
compared to several state-of-the-art approaches, especially at very low
bit-rate.

本研究提出一种基于卷积神经网络的端到端图像压缩框架，通过特征描述神经网络获取低维有效描述，借助标准图像编解码器进一步压缩图像后，再通过后处理神经网络消除失真和压缩留痕，最终通过虚拟编解码器神经网络通过反向传播优化整个系统。实验结果表明该方法在极低比特率下比当前现有算法性能更优。