We propose a) a Language Agnostic end-to-end Speech Translation model (LAST),
and b) a data augmentation strategy to increase code-switching (CS)
performance. With increasing globalization, multiple languages are increasingly
used interchangeably during fluent speech. Such CS complicates traditional
speech recognition and translation, as we must recognize which language was
spoken first and then apply a language-dependent recognizer and subsequent
translation component to generate the desired target language output. Such a
pipeline introduces latency and errors. In this paper, we eliminate the need
for that, by treating speech recognition and translation as one unified
end-to-end speech translation problem. By training LAST with both input
languages, we decode speech into one target language, regardless of the input
language. LAST delivers comparable recognition and speech translation accuracy
in monolingual usage, while reducing latency and error rate considerably when
CS is observed.

我们提出了一种语言无关的端到端语音翻译模型（LAST），并提出了一种数据增强策略来提高代码切换性能。通过使用此模型，我们可以在处理多种语言混合的自然语音时，无需使用语言特定的语音识别和翻译组件，并显著减少延迟和错误率。

无切换编码：语言无关的端到端语音翻译

Code-Switching without Switching: Language Agnostic End-to-End Speech Translation

Visual Speech Recognition (VSR) is the process of recognizing or interpreting
speech by watching the lip movements of the speaker. Recent machine learning
based approaches model VSR as a classification problem; however, the scarcity
of training data leads to error-prone systems with very low accuracies in
predicting unseen classes. To solve this problem, we present a novel approach
to zero-shot learning by generating new classes using Generative Adversarial
Networks (GANs), and show how the addition of unseen class samples increases
the accuracy of a VSR system by a significant margin of 27% and allows it to
handle speaker-independent out-of-vocabulary phrases. We also show that our
models are language agnostic and therefore capable of seamlessly generating,
using English training data, videos for a new language (Hindi). To the best of
our knowledge, this is the first work to show empirical evidence of the use of
GANs for generating training samples of unseen classes in the domain of VSR,
hence facilitating zero-shot learning. We make the added videos for new classes
publicly available along with our code.

本文介绍了一种使用生成对抗网络 (GANs) 生成新类别的零样本学习方法，用于提高视觉语音识别系统的准确性和处理跨语音的词汇。本研究展示出，该模型是语言无关的，因此可以使用英语训练数据生成新语言 (如印地语) 的视频。我们公开了添加了新类别的视频和代码。