Continual learning is crucial for dialog state tracking (DST) in dialog
systems, since requirements from users for new functionalities are often
encountered. However, most of existing continual learning methods for DST
require task identities during testing, which is a severe limit in real-world
applications. In this paper, we aim to address continual learning of DST in the
class-incremental scenario (namely the task identity is unknown in testing).
Inspired by the recently emerging prompt tuning method that performs well on
dialog systems, we propose to use the prompt pool method, where we maintain a
pool of key-value paired prompts and select prompts from the pool according to
the distance between the dialog history and the prompt keys. The proposed
method can automatically identify tasks and select appropriate prompts during
testing. We conduct experiments on Schema-Guided Dialog dataset (SGD) and
another dataset collected from a real-world dialog application. Experiment
results show that the prompt pool method achieves much higher joint goal
accuracy than the baseline. After combining with a rehearsal buffer, the model
performance can be further improved.

我们提出了一种使用提示池方法的持续学习方案，能够在测试中自动识别任务和选择适当的提示，并结合回放缓冲区进一步提高模型性能。

基于提示池的逐步增加分类的对话状态跟踪

Prompt Pool based Class-Incremental Continual Learning for Dialog State  Tracking

End-to-end task-oriented dialogue (TOD) systems have achieved promising
performance by leveraging sophisticated natural language understanding and
natural language generation capabilities of pre-trained models. This work
enables the TOD systems with more flexibility through a simple cache. The cache
provides the flexibility to dynamically update the TOD systems and handle both
existing and unseen dialogue scenarios. Towards this end, we first fine-tune a
retrieval module to effectively retrieve the most relevant information entries
from the cache. We then train end-to-end TOD models that can refer to and
ground on both dialogue history and retrieved information during TOD
generation. The cache is straightforward to construct, and the backbone models
of TOD systems are compatible with existing pre-trained generative models.
Extensive experiments demonstrate the superior performance of our framework,
with a notable improvement in non-empty joint goal accuracy by 6.7% compared to
strong baselines.

通过使用简单的缓存，该研究为任务导向对话系统带来了更大的灵活性。实验表明，该框架具有卓越的性能，与强基线相比，非空联合目标准确率提高了 6.7%。

增强检索辅助端到端面向任务系统在已见和未见对话场景中的性能

Enhancing Performance on Seen and Unseen Dialogue Scenarios using  Retrieval-Augmented End-to-End Task-Oriented System

Dialogue State Tracking (DST) is primarily evaluated using Joint Goal
Accuracy (JGA) defined as the fraction of turns where the ground-truth dialogue
state exactly matches the prediction. Generally in DST, the dialogue state or
belief state for a given turn contains all the intents shown by the user till
that turn. Due to this cumulative nature of the belief state, it is difficult
to get a correct prediction once a misprediction has occurred. Thus, although
being a useful metric, it can be harsh at times and underestimate the true
potential of a DST model. Moreover, an improvement in JGA can sometimes
decrease the performance of turn-level or non-cumulative belief state
prediction due to inconsistency in annotations. So, using JGA as the only
metric for model selection may not be ideal for all scenarios. In this work, we
discuss various evaluation metrics used for DST along with their shortcomings.
To address the existing issues, we propose a new evaluation metric named
Flexible Goal Accuracy (FGA). FGA is a generalized version of JGA. But unlike
JGA, it tries to give penalized rewards to mispredictions that are locally
correct i.e. the root cause of the error is an earlier turn. By doing so, FGA
considers the performance of both cumulative and turn-level prediction flexibly
and provides a better insight than the existing metrics. We also show that FGA
is a better discriminator of DST model performance.

本文介绍了对话状态跟踪的常用评估方法 Joint Goal Accuracy，分析了其缺陷，提出了一种新的评估指标 Flexible Goal Accuracy，该指标灵活地考虑了累计和单独分开预测的准确性，以提高对 DST 模型性能的鉴别力。

灵活吸收逐轮表现的对话状态跟踪公正评价

Towards Fair Evaluation of Dialogue State Tracking by Flexible Incorporation of Turn-level Performances

MultiWOZ is one of the most popular multi-domain task-oriented dialog
datasets, containing 10K+ annotated dialogs covering eight domains. It has been
widely accepted as a benchmark for various dialog tasks, e.g., dialog state
tracking (DST), natural language generation (NLG), and end-to-end (E2E) dialog
modeling. In this work, we identify an overlooked issue with dialog state
annotation inconsistencies in the dataset, where a slot type is tagged
inconsistently across similar dialogs leading to confusion for DST modeling. We
propose an automated correction for this issue, which is present in a whopping
70% of the dialogs. Additionally, we notice that there is significant entity
bias in the dataset (e.g., "cambridge" appears in 50% of the destination cities
in the train domain). The entity bias can potentially lead to named entity
memorization in generative models, which may go unnoticed as the test set
suffers from a similar entity bias as well. We release a new test set with all
entities replaced with unseen entities. Finally, we benchmark joint goal
accuracy (JGA) of the state-of-the-art DST baselines on these modified versions
of the data. Our experiments show that the annotation inconsistency corrections
lead to 7-10% improvement in JGA. On the other hand, we observe a 29% drop in
JGA when models are evaluated on the new test set with unseen entities.

本研究针对 MultiWOZ 数据集中 Dialog State Tracking (DST) 这一任务的注释一致性问题进行了研究，并提出了一种自动的纠正方法，通过该方法可以在数据集上进行 7-10％的 JGA 增强。此外，还检测到数据集中的实体偏差问题，并在新的测试集上测试出实体未知时，JGA 可能下降高达 29％。