Multimodal pre-training for audio-and-text has recently been proved to be
effective and has significantly improved the performance of many downstream
speech understanding tasks. However, these state-of-the-art pre-training
audio-text models work well only when provided with large amount of parallel
audio-and-text data, which brings challenges on many languages that are rich in
unimodal corpora but scarce of parallel cross-modal corpus. In this paper, we
investigate whether it is possible to pre-train an audio-text multimodal model
with extremely low-resource parallel data and extra non-parallel unimodal data.
Our pre-training framework consists of the following components: (1)
Intra-modal Denoising Auto-Encoding (IDAE), which is able to reconstruct input
text (audio) representations from a noisy version of itself. (2) Cross-modal
Denoising Auto-Encoding (CDAE), which is pre-trained to reconstruct the input
text (audio), given both a noisy version of the input text (audio) and the
corresponding translated noisy audio features (text embeddings). (3) Iterative
Denoising Process (IDP), which iteratively translates raw audio (text) and the
corresponding text embeddings (audio features) translated from previous
iteration into the new less-noisy text embeddings (audio features). We adapt a
dual cross-modal Transformer as our backbone model which consists of two
unimodal encoders for IDAE and two cross-modal encoders for CDAE and IDP. Our
method achieves comparable performance on multiple downstream speech
understanding tasks compared with the model pre-trained on fully parallel data,
demonstrating the great potential of the proposed method. Our code is available
at: https://github.com/KarlYuKang/Low-Resource-Multimodal-Pre-training.

本文旨在探讨在极低的跨模态数据情况下，通过利用单模态数据和翻译噪声特征进行训练，达到音频 - 文本模态的预训练效果，并证明本方法在很多语言上的表现与全并行语音 - 文本预训练数据相当。

极低资源并行数据下的自监督音频文本预训练

Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource  Parallel Data

Recently, universal neural machine translation (NMT) with shared
encoder-decoder gained good performance on zero-shot translation. Unlike
universal NMT, jointly trained language-specific encoders-decoders aim to
achieve universal representation across non-shared modules, each of which is
for a language or language family. The non-shared architecture has the
advantage of mitigating internal language competition, especially when the
shared vocabulary and model parameters are restricted in their size. However,
the performance of using multiple encoders and decoders on zero-shot
translation still lags behind universal NMT. In this work, we study zero-shot
translation using language-specific encoders-decoders. We propose to generalize
the non-shared architecture and universal NMT by differentiating the
Transformer layers between language-specific and interlingua. By selectively
sharing parameters and applying cross-attentions, we explore maximizing the
representation universality and realizing the best alignment of
language-agnostic information. We also introduce a denoising auto-encoding
(DAE) objective to jointly train the model with the translation task in a
multi-task manner. Experiments on two public multilingual parallel datasets
show that our proposed model achieves a competitive or better results than
universal NMT and strong pivot baseline. Moreover, we experiment incrementally
adding new language to the trained model by only updating the new model
parameters. With this little effort, the zero-shot translation between this
newly added language and existing languages achieves a comparable result with
the model trained jointly from scratch on all languages.

本文介绍了一种利用语言特定的编码器 - 解码器完成零样本翻译的方法。通过区分语言特定的 Transformer 层和公共层，应用交叉注意力和参数共享等技术，最大化句子的普适性，以实现最佳的语义对齐，此外结合了自编码目标来实现多任务的联合训练。在两个公共的多语言平行数据集上进行实验，它们的结果具有竞争力，而且相当于从头开始联合训练模型时的结果，还表现出增量学习的能力。