Multilingual machine translation (MMT), trained on a mixture of parallel and
monolingual data, is key for improving translation in low-resource language
pairs. However, the literature offers conflicting results on the performance of
different methods. To resolve this, we examine how denoising autoencoding (DAE)
and backtranslation (BT) impact MMT under different data conditions and model
scales. Unlike prior studies, we use a realistic dataset of 100 directions and
consider many domain combinations of monolingual and test data. We find that
monolingual data generally helps MMT, but models are surprisingly brittle to
domain mismatches, especially at smaller model scales. BT is beneficial when
the parallel, monolingual, and test data sources are similar but can be
detrimental otherwise, while DAE is less effective than previously reported.
Next, we analyze the impact of scale (from 90M to 1.6B parameters) and find it
is important for both methods, particularly DAE. As scale increases, DAE
transitions from underperforming the parallel-only baseline at 90M to
converging with BT performance at 1.6B, and even surpassing it in low-resource.
These results offer new insights into how to best use monolingual data in MMT.

通过实验研究了多种语言机器翻译在不同数据条件和模型尺度下，去噪自编码器和回译技术对多种语言机器翻译的影响、以及单语数据在多种语言机器翻译中的作用和数据规模对机器翻译的影响等问题。结果发现单语数据总体有益于机器翻译，但在小型模型中对领域不匹配，BT 对相似的平行，单语和测试数据来源是有益的，但在其他情况下会有不利影响，DAE 相对于以前报告的效果不如。规模永远是关键。

单语数据何时对多语言翻译有帮助：领域和模型规模的作用

When Does Monolingual Data Help Multilingual Translation: The Role of  Domain and Model Scale

Recent extensions of Cellular Automata (CA) have incorporated key ideas from
modern deep learning, dramatically extending their capabilities and catalyzing
a new family of Neural Cellular Automata (NCA) techniques. Inspired by
Transformer-based architectures, our work presents a new class of
$\textit{attention-based}$ NCAs formed using a spatially
localized$\unicode{x2014}$yet globally organized$\unicode{x2014}$self-attention
scheme. We introduce an instance of this class named $\textit{Vision
Transformer Cellular Automata}$ (ViTCA). We present quantitative and
qualitative results on denoising autoencoding across six benchmark datasets,
comparing ViTCA to a U-Net, a U-Net-based CA baseline (UNetCA), and a Vision
Transformer (ViT). When comparing across architectures configured to similar
parameter complexity, ViTCA architectures yield superior performance across all
benchmarks and for nearly every evaluation metric. We present an ablation study
on various architectural configurations of ViTCA, an analysis of its effect on
cell states, and an investigation on its inductive biases. Finally, we examine
its learned representations via linear probes on its converged cell state
hidden representations, yielding, on average, superior results when compared to
our U-Net, ViT, and UNetCA baselines.

本文介绍了一种名为 Vision Transformer Cellular Automata（ViTCA）的定义在细胞自动机框架内基于自注意力机制的人工神经网络，并比较了其在线性降噪自编码任务上与其他神经网络的性能。