Despite the recent advances in open-domain dialogue systems, building a
reliable evaluation metric is still a challenging problem. Recent studies
proposed learnable metrics based on classification models trained to
distinguish the correct response. However, neural classifiers are known to make
overly confident predictions for examples from unseen distributions. We propose
DEnsity, which evaluates a response by utilizing density estimation on the
feature space derived from a neural classifier. Our metric measures how likely
a response would appear in the distribution of human conversations. Moreover,
to improve the performance of DEnsity, we utilize contrastive learning to
further compress the feature space. Experiments on multiple response evaluation
datasets show that DEnsity correlates better with human evaluations than the
existing metrics. Our code is available at this https URL

本文提出了一种名为 DEnsity 的评估指标，该指标通过利用神经分类器导出的特征空间上的密度估计来评估响应，从而使得其相关性比现有指标更好，并通过对特征空间的压缩利用对比学习来进一步提高性能。

DEnsity：利用密度估计的开放领域对话评估指标

DEnsity: Open-domain Dialogue Evaluation Metric using Density Estimation

Training learnable metrics using modern language models has recently emerged
as a promising method for the automatic evaluation of machine translation.
However, existing human evaluation datasets in text simplification are limited
by a lack of annotations, unitary simplification types, and outdated models,
making them unsuitable for this approach. To address these issues, we introduce
the SIMPEVAL corpus that contains: SIMPEVAL_ASSET, comprising 12K human ratings
on 2.4K simplifications of 24 systems, and SIMPEVAL_2022, a challenging
simplification benchmark consisting of over 1K human ratings of 360
simplifications including generations from GPT-3.5. Training on SIMPEVAL_ASSET,
we present LENS, a Learnable Evaluation Metric for Text Simplification.
Extensive empirical results show that LENS correlates better with human
judgment than existing metrics, paving the way for future progress in the
evaluation of text simplification. To create the SIMPEVAL datasets, we
introduce RANK & RATE, a human evaluation framework that rates simplifications
from several models in a list-wise manner by leveraging an interactive
interface, which ensures both consistency and accuracy in the evaluation
process. Our metric, dataset, and annotation toolkit are available at
this https URL.

该论文介绍了使用 SIMPEVAL 语料库来训练可学习度量，并基于此提出了针对文本简化的可学习评估度量 LENS，结果显示与现有度量相比 LENS 更符合人类评价标准，同时提供了人类评估框架及评估工具包。