In this work, we evaluate various existing dialogue relevance metrics, find
strong dependency on the dataset, often with poor correlation with human scores
of relevance, and propose modifications to reduce data requirements and domain
sensitivity while improving correlation. Our proposed metric achieves
state-of-the-art performance on the HUMOD dataset while reducing measured
sensitivity to dataset by 37%-66%. We achieve this without fine-tuning a
pretrained language model, and using only 3,750 unannotated human dialogues and
a single negative example. Despite these limitations, we demonstrate
competitive performance on four datasets from different domains. Our code,
including our metric and experiments, is open sourced.

评估了现有的对话相关性度量标准，提出了减少数据要求和领域敏感性的改进并在不 fine-tuning 预训练语言模型的情况下，只使用 3750 个未经注释的人类对话和一个负面例子，我们的提议指标实现了 HUMOD dataset 的最新性能，并将数据集敏感性降低了 37％-66％。