The Universal Transformer (UT) is a variant of the Transformer that shares
parameters across its layers. Empirical evidence shows that UTs have better
compositional generalization than Vanilla Transformers (VTs) in formal language
tasks. The parameter-sharing also affords it better parameter efficiency than
VTs. Despite its many advantages, scaling UT parameters is much more compute
and memory intensive than scaling up a VT. This paper proposes the Sparse
Universal Transformer (SUT), which leverages Sparse Mixture of Experts (SMoE)
and a new stick-breaking-based dynamic halting mechanism to reduce UT's
computation complexity while retaining its parameter efficiency and
generalization ability. Experiments show that SUT achieves the same performance
as strong baseline models while only using half computation and parameters on
WMT'14 and strong generalization results on formal language tasks (Logical
inference and CFQ). The new halting mechanism also enables around 50\%
reduction in computation during inference with very little performance decrease
on formal language tasks.

提出了 Sparse Universal Transformer (SUT)，利用稀疏专家混合（SMoE）和基于断裂棒的动态终止机制来减少 UT 的计算复杂性，同时保持其参数效率和泛化能力。实验证明，SUT 在 WMT'14 上仅使用一半的计算和参数与强基准模型实现相同性能，并在形式语言任务（逻辑推理和 CFQ）上获得强大的泛化结果。新的终止机制还在推理过程中实现了大约 50% 的计算减少，而在形式语言任务上的性能降低非常小。

稀疏通用变压器

Sparse Universal Transformer

The article is focused on automatic development and ranking of a large corpus
for Russian paraphrase generation which proves to be the first corpus of such
type in Russian computational linguistics. Existing manually annotated
paraphrase datasets for Russian are limited to small-sized ParaPhraser corpus
and ParaPlag which are suitable for a set of NLP tasks, such as paraphrase and
plagiarism detection, sentence similarity and relatedness estimation, etc. Due
to size restrictions, these datasets can hardly be applied in end-to-end text
generation solutions. Meanwhile, paraphrase generation requires a large amount
of training data. In our study we propose a solution to the problem: we
collect, rank and evaluate a new publicly available headline paraphrase corpus
(ParaPhraser Plus), and then perform text generation experiments with manual
evaluation on automatically ranked corpora using the Universal Transformer
architecture.

本文是关于用于俄语释义生成的大规模语料库进行自动开发和排名的，提出了 ParaPhraser Plus 数据集以解决目前该领域小规模的语料库难以在文本生成中应用的问题，并使用 Universal Transformer 架构进行了实验。

自动生成排名的俄语改述语料库用于文本生成

Automatically Ranked Russian Paraphrase Corpus for Text Generation

Recurrent neural networks (RNNs) sequentially process data by updating their
state with each new data point, and have long been the de facto choice for
sequence modeling tasks. However, their inherently sequential computation makes
them slow to train. Feed-forward and convolutional architectures have recently
been shown to achieve superior results on some sequence modeling tasks such as
machine translation, with the added advantage that they concurrently process
all inputs in the sequence, leading to easy parallelization and faster training
times. Despite these successes, however, popular feed-forward sequence models
like the Transformer fail to generalize in many simple tasks that recurrent
models handle with ease, e.g. copying strings or even simple logical inference
when the string or formula lengths exceed those observed at training time. We
propose the Universal Transformer (UT), a parallel-in-time self-attentive
recurrent sequence model which can be cast as a generalization of the
Transformer model and which addresses these issues. UTs combine the
parallelizability and global receptive field of feed-forward sequence models
like the Transformer with the recurrent inductive bias of RNNs. We also add a
dynamic per-position halting mechanism and find that it improves accuracy on
several tasks. In contrast to the standard Transformer, under certain
assumptions, UTs can be shown to be Turing-complete. Our experiments show that
UTs outperform standard Transformers on a wide range of algorithmic and
language understanding tasks, including the challenging LAMBADA language
modeling task where UTs achieve a new state of the art, and machine translation
where UTs achieve a 0.9 BLEU improvement over Transformers on the WMT14 En-De
dataset.

Universal Transformer 是一种并行自我关注循环序列模型，它将前馈序列模型（例如 Transformer）的并行性和全局感受野与递归的归纳偏差相结合，解决了在某些简单任务中标准 Transformer 失败的问题，并取得诸多算法和语言理解任务上优越的表现。