The Universal Transformer (UT) is a variant of the Transformer that shares
parameters across its layers. Empirical evidence shows that UTs have better
compositional generalization than Vanilla Transformers (VTs) in formal language
tasks. The parameter-sharing also affords it better parameter efficiency than
VTs. Despite its many advantages, scaling UT parameters is much more compute
and memory intensive than scaling up a VT. This paper proposes the Sparse
Universal Transformer (SUT), which leverages Sparse Mixture of Experts (SMoE)
and a new stick-breaking-based dynamic halting mechanism to reduce UT's
computation complexity while retaining its parameter efficiency and
generalization ability. Experiments show that SUT achieves the same performance
as strong baseline models while only using half computation and parameters on
WMT'14 and strong generalization results on formal language tasks (Logical
inference and CFQ). The new halting mechanism also enables around 50\%
reduction in computation during inference with very little performance decrease
on formal language tasks.

提出了 Sparse Universal Transformer (SUT)，利用稀疏专家混合（SMoE）和基于断裂棒的动态终止机制来减少 UT 的计算复杂性，同时保持其参数效率和泛化能力。实验证明，SUT 在 WMT'14 上仅使用一半的计算和参数与强基准模型实现相同性能，并在形式语言任务（逻辑推理和 CFQ）上获得强大的泛化结果。新的终止机制还在推理过程中实现了大约 50% 的计算减少，而在形式语言任务上的性能降低非常小。