Given a small corpus $\mathcal D_T$ pertaining to a limited set of focused
topics, our goal is to train embeddings that accurately capture the sense of
words in the topic in spite of the limited size of $\mathcal D_T$. These
embeddings may be used in various tasks involving $\mathcal D_T$. A popular
strategy in limited data settings is to adapt pre-trained embeddings $\mathcal
E$ trained on a large corpus. To correct for sense drift, fine-tuning,
regularization, projection, and pivoting have been proposed recently. Among
these, regularization informed by a word's corpus frequency performed well, but
we improve upon it using a new regularizer based on the stability of its
cooccurrence with other words. However, a thorough comparison across ten
topics, spanning three tasks, with standardized settings of hyper-parameters,
reveals that even the best embedding adaptation strategies provide small gains
beyond well-tuned baselines, which many earlier comparisons ignored. In a bold
departure from adapting pretrained embeddings, we propose using $\mathcal D_T$
to probe, attend to, and borrow fragments from any large, topic-rich source
corpus (such as Wikipedia), which need not be the corpus used to pretrain
embeddings. This step is made scalable and practical by suitable indexing. We
reach the surprising conclusion that even limited corpus augmentation is more
useful than adapting embeddings, which suggests that non-dominant sense
information may be irrevocably obliterated from pretrained embeddings and
cannot be salvaged by adaptation.

本文旨在通过训练嵌入来准确捕获少量数据集上限制主题的单词的意义，并可能在各种涉及该数据集的任务中使用它们。为了适应预先训练的嵌入，提出了一些策略，其中包括基于单词语料库频率的正则化。不过，本文提出的一种新的稳定度正则化策略表现更加优秀，这种策略在少量数据语境中很实用。通过在十个主题上的三个任务中的比较，我们发现，即使是最佳的嵌入适配策略也只能提供比微调基线略微增益的结果。最后，本文提出的非常见的预处理嵌入方式是通过从大型主题丰富的源语料库 (如维基百科) 中探索、关注和借用片段来实现的，这一步骤是通过合适的索引方法实现的。通过实验，本文得出了一个惊人的结论，即即使是很少的语料库增强也比调整嵌入更有用，这表明习惯性的语境信息可能已经从预训练的嵌入中消失，不能通过适配来挽救。

通用语料库的主题敏感注意力机制纠正预训练嵌入中的词义偏差

Topic Sensitive Attention on Generic Corpora Corrects Sense Bias in  Pretrained Embeddings

We investigate end-to-end speech-to-text translation on a corpus of
audiobooks specifically augmented for this task. Previous works investigated
the extreme case where source language transcription is not available during
learning nor decoding, but we also study a midway case where source language
transcription is available at training time only. In this case, a single model
is trained to decode source speech into target text in a single pass.
Experimental results show that it is possible to train compact and efficient
end-to-end speech translation models in this setup. We also distribute the
corpus and hope that our speech translation baseline on this corpus will be
challenged in the future.

本研究旨在探讨在经过语料库特殊增强的情况下，对语音到文本翻译进行端到端的研究。我们不仅研究了在学习和解码期间没有源语言转录的极端情况，还研究了在训练时仅提供源语言转录的中间情况。实验结果表明，在这种设置下，可以训练出紧凑高效的端到端语音转换模型。同时，我们提供了语料库，并希望未来的研究能够挑战我们在该语料库上的语音翻译基线模型。