This work proposes a novel Graph-based neural ArchiTecture Encoding Scheme,
a.k.a. GATES, to improve the predictor-based neural architecture search.
Specifically, different from existing graph-based schemes, GATES models the
operations as the transformation of the propagating information, which mimics
the actual data processing of neural architecture. GATES is a more reasonable
modeling of the neural architectures, and can encode architectures from both
the "operation on node" and "operation on edge" cell search spaces
consistently. Experimental results on various search spaces confirm GATES's
effectiveness in improving the performance predictor. Furthermore, equipped
with the improved performance predictor, the sample efficiency of the
predictor-based neural architecture search (NAS) flow is boosted. Codes are
available at this https URL

提出一种新型的基于图的神经架构编码方案 GATES，通过模拟神经架构的实际数据处理方式 -- 传播信息的转换，改进了基于预测器的神经架构搜索，并成功地提高了搜索样本的效率。

基于图神经网络的通用架构编码方案，用于基于预测的 NAS

A Generic Graph-based Neural Architecture Encoding Scheme for  Predictor-based NAS

An effective method to generate a large number of parallel sentences for
training improved neural machine translation (NMT) systems is the use of the
back-translations of the target-side monolingual data. The standard
back-translation method has been shown to be unable to efficiently utilize the
available huge amount of existing monolingual data because of the inability of
translation models to differentiate between the authentic and synthetic
parallel data during training. Tagging, or using gates, has been used to enable
translation models to distinguish between synthetic and authentic data,
improving standard back-translation and also enabling the use of iterative
back-translation on language pairs that underperformed using standard
back-translation. In this work, we approach back-translation as a domain
adaptation problem, eliminating the need for explicit tagging. In the approach
-- \emph{tag-less back-translation} -- the synthetic and authentic parallel
data are treated as out-of-domain and in-domain data respectively and, through
pre-training and fine-tuning, the translation model is shown to be able to
learn more efficiently from them during training. Experimental results have
shown that the approach outperforms the standard and tagged back-translation
approaches on low resource English-Vietnamese and English-German neural machine
translation.

研究发现，采用无标记的反向翻译方法可以有效地从大量单语数据中生成平行句子，进而提高神经机器翻译的效果。该方法能够消除标记语境的需求，并在低资源的英越和英德神经机器翻译任务上表现出比标准方法和标记方法更好的性能。

无标签回译

Tag-less Back-Translation

Long Short-Term Memory (LSTM) is one of the most widely used recurrent
structures in sequence modeling. It aims to use gates to control information
flow (e.g., whether to skip some information or not) in the recurrent
computations, although its practical implementation based on soft gates only
partially achieves this goal. In this paper, we propose a new way for LSTM
training, which pushes the output values of the gates towards 0 or 1. By doing
so, we can better control the information flow: the gates are mostly open or
closed, instead of in a middle state, which makes the results more
interpretable. Empirical studies show that (1) Although it seems that we
restrict the model capacity, there is no performance drop: we achieve better or
comparable performances due to its better generalization ability; (2) The
outputs of gates are not sensitive to their inputs: we can easily compress the
LSTM unit in multiple ways, e.g., low-rank approximation and low-precision
approximation. The compressed models are even better than the baseline models
without compression.

本研究提出了一种新的 LSTM 训练方式，使得门控单元的输出更加容易解释，经过实证研究，发现通过将门控单元的输出值推向 0 或 1，可以更好地控制信息流，从而提高模型的泛化能力和压缩率。

朝向二值门的鲁棒 LSTM 训练

Towards Binary-Valued Gates for Robust LSTM Training

We propose a practical method for $L_0$ norm regularization for neural
networks: pruning the network during training by encouraging weights to become
exactly zero. Such regularization is interesting since (1) it can greatly speed
up training and inference, and (2) it can improve generalization. AIC and BIC,
well-known model selection criteria, are special cases of $L_0$ regularization.
However, since the $L_0$ norm of weights is non-differentiable, we cannot
incorporate it directly as a regularization term in the objective function. We
propose a solution through the inclusion of a collection of non-negative
stochastic gates, which collectively determine which weights to set to zero. We
show that, somewhat surprisingly, for certain distributions over the gates, the
expected $L_0$ norm of the resulting gated weights is differentiable with
respect to the distribution parameters. We further propose the \emph{hard
concrete} distribution for the gates, which is obtained by "stretching" a
binary concrete distribution and then transforming its samples with a
hard-sigmoid. The parameters of the distribution over the gates can then be
jointly optimized with the original network parameters. As a result our method
allows for straightforward and efficient learning of model structures with
stochastic gradient descent and allows for conditional computation in a
principled way. We perform various experiments to demonstrate the effectiveness
of the resulting approach and regularizer.

本研究提出了一种使用稀疏性约束进行神经网络剪枝的方法，该方法通过一系列随机门来收缩网络，以便训练和预测运算可以更加快速和高效。