Minimum Bayes-risk (MBR) decoding has recently gained renewed attention in
text generation. MBR decoding considers texts sampled from a model as
pseudo-references and selects the text with the highest similarity to the
others. Therefore, sampling is one of the key elements of MBR decoding, and
previous studies reported that the performance varies by sampling methods. From
a theoretical standpoint, this performance variation is likely tied to how
closely the samples approximate the true distribution of references. However,
this approximation has not been the subject of in-depth study. In this study,
we propose using anomaly detection to measure the degree of approximation. We
first closely examine the performance variation and then show that previous
hypotheses about samples do not correlate well with the variation, but our
introduced anomaly scores do. The results are the first to empirically support
the link between the performance and the core assumption of MBR decoding.

使用异常检测来衡量最小贝叶斯风险解码的核心假设与性能之间的联系。

关于最小贝叶斯风险解码真实分布逼近

On the True Distribution Approximation of Minimum Bayes-Risk Decoding

Large language models (LLMs) are becoming a one-fits-many solution, but they
sometimes hallucinate or produce unreliable output. In this paper, we
investigate how hypothesis ensembling can improve the quality of the generated
text for the specific problem of LLM-based machine translation. We experiment
with several techniques for ensembling hypotheses produced by LLMs such as
ChatGPT, LLaMA, and Alpaca. We provide a comprehensive study along multiple
dimensions, including the method to generate hypotheses (multiple prompts,
temperature-based sampling, and beam search) and the strategy to produce the
final translation (instruction-based, quality-based reranking, and minimum
Bayes risk (MBR) decoding). Our results show that MBR decoding is a very
effective method, that translation quality can be improved using a small number
of samples, and that instruction tuning has a strong impact on the relation
between the diversity of the hypotheses and the sampling temperature.

通过对多个大型语言模型 (LLMs) 进行假设集成的研究，对于基于 LLM 的机器翻译问题，我们探讨了如何提高生成文本的质量。我们尝试了多种假设集成技术，如 ChatGPT、LLaMA 和 Alpaca，并进行了综合研究，包括生成假设的方法（多个提示、温度采样和束搜索）以及生成最终翻译的策略（基于指令、基于质量的重新排序和最小贝叶斯风险解码）。我们的研究结果表明，MBR 解码是一种非常有效的方法，使用少量样本可以提高翻译质量，指令调整对假设的多样性和采样温度之间的关系具有很大影响。

使用大型语言模型进行翻译假设集成的实证研究

An Empirical Study of Translation Hypothesis Ensembling with Large  Language Models

Minimum Bayes Risk (MBR) decoding is a method for choosing the outputs of a
machine learning system based not on the output with the highest probability,
but the output with the lowest risk (expected error) among multiple candidates.
It is a simple but powerful method: for an additional cost at inference time,
MBR provides reliable several-point improvements across metrics for a wide
variety of tasks without any additional data or training. Despite this, MBR is
not frequently applied in NLP works, and knowledge of the method itself is
limited. We first provide an introduction to the method and the recent
literature. We show that several recent methods that do not reference MBR can
be written as special cases of MBR; this reformulation provides additional
theoretical justification for the performance of these methods, explaining some
results that were previously only empirical. We provide theoretical and
empirical results about the effectiveness of various MBR variants and make
concrete recommendations for the application of MBR in NLP models, including
future directions in this area.

最小贝叶斯风险（MBR）解码是一种选择机器学习系统输出的方法，不是基于最高概率的输出，而是基于多个候选项中具有最低风险（预期误差）的输出。本文首先介绍了该方法和最近的文献，证明了一些最近的方法可以被写成 MBR 的特例，为这些方法的性能提供了额外的理论依据，并给出了在 NLP 模型中应用 MBR 的理论和实证结果，包括该领域的未来方向的具体建议。

MBR 从始至终：通过极小贝叶斯风险的视角探讨现代生成技术

It's MBR All the Way Down: Modern Generation Techniques Through the Lens  of Minimum Bayes Risk

We present the CUNI-Bergamot submission for the WMT22 General translation
task. We compete in English$\rightarrow$Czech direction. Our submission further
explores block backtranslation techniques. Compared to the previous work, we
measure performance in terms of COMET score and named entities translation
accuracy. We evaluate performance of MBR decoding compared to traditional mixed
backtranslation training and we show a possible synergy when using both of the
techniques simultaneously. The results show that both approaches are effective
means of improving translation quality and they yield even better results when
combined.

CUNI-Bergamot 参加 WMT22 普通翻译任务，通过分析 COMET 分数和命名实体翻译准确性，比较传统混合回译训练和 MBR 解码的性能，结果表明两种方法都有效地提高了翻译质量，并且当同时应用时效果更好。