Despite the known limitations, most machine translation systems today still
operate on the sentence-level. One reason for this is, that most parallel
training data is only sentence-level aligned, without document-level meta
information available. In this work, we set out to build context-aware
translation systems utilizing document-level monolingual data instead. This can
be achieved by combining any existing sentence-level translation model with a
document-level language model. We improve existing approaches by leveraging
recent advancements in model combination. Additionally, we propose novel
weighting techniques that make the system combination more flexible and
significantly reduce computational overhead. In a comprehensive evaluation on
four diverse translation tasks, we show that our extensions improve
document-targeted scores substantially and are also computationally more
efficient. However, we also find that in most scenarios, back-translation gives
even better results, at the cost of having to re-train the translation system.
Finally, we explore language model fusion in the light of recent advancements
in large language models. Our findings suggest that there might be strong
potential in utilizing large language models via model combination.

本研究针对机器翻译系统在句子级别上操作的限制，提出了利用文档级单语数据构建上下文感知翻译系统的方法，并通过结合现有句子级翻译模型和文档级语言模型，改进了现有方法，并提出了新的权重技术以提高系统组合的灵活性和降低计算开销。通过对四个不同的翻译任务进行全面评估，结果显示我们的改进能够显著提高面向文档的评分，并在计算上更高效。但在大多数情况下，通过再训练翻译系统进行反向翻译可以获得更好的结果。最后，我们探索了基于大语言模型的语言模型融合，在最新的大语言模型进展的基础上取得了一定的潜力。

机器翻译中的文档级语言模型

Document-Level Language Models for Machine Translation

Language model fusion helps smart assistants recognize words which are rare
in acoustic data but abundant in text-only corpora (typed search logs).
However, such corpora have properties that hinder downstream performance,
including being (1) too large, (2) beset with domain-mismatched content, and
(3) heavy-headed rather than heavy-tailed (excessively many duplicate search
queries such as "weather"). We show that three simple strategies for selecting
language modeling data can dramatically improve rare-word recognition without
harming overall performance. First, to address the heavy-headedness, we
downsample the data according to a soft log function, which tunably reduces
high frequency (head) sentences. Second, to encourage rare-word exposure, we
explicitly filter for words rare in the acoustic data. Finally, we tackle
domain-mismatch via perplexity-based contrastive selection, filtering for
examples matched to the target domain. We down-select a large corpus of web
search queries by a factor of 53x and achieve better LM perplexities than
without down-selection. When shallow-fused with a state-of-the-art, production
speech engine, our LM achieves WER reductions of up to 24% relative on
rare-word sentences (without changing overall WER) compared to a baseline LM
trained on the raw corpus. These gains are further validated through favorable
side-by-side evaluations on live voice search traffic.

通过降采样、明确筛选稀有词以及使用基于困惑度的对比筛选等简单的策略，结合生产语音引擎，利用语言模型融合技术，相对于使用原始语料训练出的语言模型，使智能语音助手能够更好地辨别听众说出的稀有词而不影响总体识别准确率，并在实时语音搜索流量中得到了有利的侧面对比评估结果。