Recently, a human evaluation study of Referring Expression Generation (REG)
models had an unexpected conclusion: on \textsc{webnlg}, Referring Expressions
(REs) generated by the state-of-the-art neural models were not only
indistinguishable from the REs in \textsc{webnlg} but also from the REs
generated by a simple rule-based system. Here, we argue that this limitation
could stem from the use of a purely ratings-based human evaluation (which is a
common practice in Natural Language Generation). To investigate these issues,
we propose an intrinsic task-based evaluation for REG models, in which, in
addition to rating the quality of REs, participants were asked to accomplish
two meta-level tasks. One of these tasks concerns the referential success of
each RE; the other task asks participants to suggest a better alternative for
each RE. The outcomes suggest that, in comparison to previous evaluations, the
new evaluation protocol assesses the performance of each REG model more
comprehensively and makes the participants' ratings more reliable and
discriminable.

提出了一种基于内在任务的评估协议，针对对话生成模型中的表述生成模型进行了综合性评估，发现新的评估协议能更准确和可靠地评估每个模型的性能。

指代表达生成的内在任务评估

Intrinsic Task-based Evaluation for Referring Expression Generation

As a first step towards agents learning to communicate about their visual
environment, we propose a system that, given visual representations of a
referent (cat) and a context (sofa), identifies their discriminative
attributes, i.e., properties that distinguish them (has_tail). Moreover,
despite the lack of direct supervision at the attribute level, the model learns
to assign plausible attributes to objects (sofa-has_cushion). Finally, we
present a preliminary experiment confirming the referential success of the
predicted discriminative attributes.

研究了代理人在视觉环境中学习进行通讯的第一步，通过给定参照物（猫）和环境（沙发）的视觉表示，确定它们的可区分属性，即区分它们的属性（有尾巴）；此外，尽管缺乏对属性级别的直接监督，模型还学会了为对象分配有合理的属性（沙发 - 有靠垫）；最后，通过一个初步实验确认了预测的可区分属性的参考成功。