This paper introduces an adversarial method to stress-test trained metrics to
evaluate conversational dialogue systems. The method leverages Reinforcement
Learning to find response strategies that elicit optimal scores from the
trained metrics. We apply our method to test recently proposed trained metrics.
We find that they all are susceptible to giving high scores to responses
generated by relatively simple and obviously flawed strategies that our method
converges on. For instance, simply copying parts of the conversation context to
form a response yields competitive scores or even outperforms responses written
by humans.

本文介绍一种对话系统对话回复的鲁棒性测试方法，利用对抗学习方法提取出优化得分的回复策略，并利用该方法测试最近提出的训练度量标准，发现它们均容易将相对简单且明显存在缺陷的策略高分，如直接复制对话环境中的部分内容拼成回复竟然可以超越甚至优于人类翻译的水平。

探究面向对话系统的训练度量的鲁棒性

Probing the Robustness of Trained Metrics for Conversational Dialogue Systems

Recent developments in machine translation and multilingual text generation
have led researchers to adopt trained metrics such as COMET or BLEURT, which
treat evaluation as a regression problem and use representations from
multilingual pre-trained models such as XLM-RoBERTa or mBERT. Yet studies on
related tasks suggest that these models are most efficient when they are large,
which is costly and impractical for evaluation. We investigate the trade-off
between multilinguality and model capacity with RemBERT, a state-of-the-art
multilingual language model, using data from the WMT Metrics Shared Task. We
present a series of experiments which show that model size is indeed a
bottleneck for cross-lingual transfer, then demonstrate how distillation can
help addressing this bottleneck, by leveraging synthetic data generation and
transferring knowledge from one teacher to multiple students trained on related
languages. Our method yields up to 10.5% improvement over vanilla fine-tuning
and reaches 92.6% of RemBERT's performance using only a third of its
parameters.

使用 RemBERT 模型的实验表明，模型大小限制了跨语言转换的效率，将知识从一名教师传递到以相关语言训练的多个学生可以通过蒸馏解决这个问题，该方法可以使性能提高 10.5％，并仅使用 RemBERT 的三分之一的参数即可达到 92.6％的性能。