Ensuring that generated utterances are faithful to dialogue actions is
crucial for Task-Oriented Dialogue Response Generation. Slot Error Rate (SER)
only partially measures generation quality in that it solely assesses
utterances generated from non-categorical slots whose values are expected to be
reproduced exactly. Utterances generated from categorical slots, which are more
variable, are not assessed by SER. We propose Schema-Guided Semantic Accuracy
(SGSAcc) to evaluate utterances generated from both categorical and
non-categorical slots by recognizing textual entailment. We show that SGSAcc
can be applied to evaluate utterances generated from a wide range of dialogue
actions in the Schema Guided Dialogue (SGD) dataset with good agreement with
human judgment. We also identify a previously overlooked weakness in generating
faithful utterances from categorical slots in unseen domains. We show that
prefix tuning applied to T5 generation can address this problem. We further
build an ensemble of prefix-tuning and fine-tuning models that achieves the
lowest SER reported and high SGSAcc on the SGD dataset.

这篇研究论文探讨了如何提高面向任务的对话回应生成中生成回应的质量评估问题，并提出了一种新的评估方法：基于模式的语义准确性评估，并利用预训练模型的前缀调优技术解决了在生成非分类槽的回应时的困难，得到了在不同对话场景下较好的评估结果。

基于模式引导的语义准确性：面向任务导向的对话响应生成中的正确性

Schema-Guided Semantic Accuracy: Faithfulness in Task-Oriented Dialogue Response Generation

Moving from limited-domain natural language generation (NLG) to open domain
is difficult because the number of semantic input combinations grows
exponentially with the number of domains. Therefore, it is important to
leverage existing resources and exploit similarities between domains to
facilitate domain adaptation. In this paper, we propose a procedure to train
multi-domain, Recurrent Neural Network-based (RNN) language generators via
multiple adaptation steps. In this procedure, a model is first trained on
counterfeited data synthesised from an out-of-domain dataset, and then fine
tuned on a small set of in-domain utterances with a discriminative objective
function. Corpus-based evaluation results show that the proposed procedure can
achieve competitive performance in terms of BLEU score and slot error rate
while significantly reducing the data needed to train generators in new, unseen
domains. In subjective testing, human judges confirm that the procedure greatly
improves generator performance when only a small amount of data is available in
the domain.

本研究提供了一种训练多领域、基于递归神经网络的语言生成器的程序，包含了多个适应步骤，并利用伪造数据和领域内句子的优化目标函数，旨在在新的、看不见的领域内显著减少培训的数据需求，其语料库评估结果和主观测试都表示该程序可以在保证 BLEU 分数和槽误差率竞争力的同时大幅提高语言生成器性能。