Conversational question answering aims to provide natural-language answers to
users in information-seeking conversations. Existing conversational QA
benchmarks compare models with pre-collected human-human conversations, using
ground-truth answers provided in conversational history. It remains unclear
whether we can rely on this static evaluation for model development and whether
current systems can well generalize to real-world human-machine conversations.
In this work, we conduct the first large-scale human evaluation of
state-of-the-art conversational QA systems, where human evaluators converse
with models and judge the correctness of their answers. We find that the
distribution of human machine conversations differs drastically from that of
human-human conversations, and there is a disagreement between human and
gold-history evaluation in terms of model ranking. We further investigate how
to improve automatic evaluations, and propose a question rewriting mechanism
based on predicted history, which better correlates with human judgments.
Finally, we analyze the impact of various modeling strategies and discuss
future directions towards building better conversational question answering
systems.

本文对目前最先进的对话问答系统进行了首次的大规模人类评估，发现人类与机器之间的对话存在显著差异。此外，基于预测历史的问题重写机制能够更好地与人类判断相一致。通过研究各种建模策略，讨论了构建更好的对话问答系统的未来方向。

去除黄金标准：重新评估对话问答

Ditch the Gold Standard: Re-evaluating Conversational Question Answering

Classifying the general intent of the user utterance in a conversation, also
known as Dialogue Act (DA), e.g., open-ended question, statement of opinion, or
request for an opinion, is a key step in Natural Language Understanding (NLU)
for conversational agents. While DA classification has been extensively studied
in human-human conversations, it has not been sufficiently explored for the
emerging open-domain automated conversational agents. Moreover, despite
significant advances in utterance-level DA classification, full understanding
of dialogue utterances requires conversational context. Another challenge is
the lack of available labeled data for open-domain human-machine conversations.
To address these problems, we propose a novel method, CDAC (Contextual Dialogue
Act Classifier), a simple yet effective deep learning approach for contextual
dialogue act classification. Specifically, we use transfer learning to adapt
models trained on human-human conversations to predict dialogue acts in
human-machine dialogues. To investigate the effectiveness of our method, we
train our model on the well-known Switchboard human-human dialogue dataset, and
fine-tune it for predicting dialogue acts in human-machine conversation data,
collected as part of the Amazon Alexa Prize 2018 competition. The results show
that the CDAC model outperforms an utterance-level state of the art baseline by
8.0% on the Switchboard dataset, and is comparable to the latest reported
state-of-the-art contextual DA classification results. Furthermore, our results
show that fine-tuning the CDAC model on a small sample of manually labeled
human-machine conversations allows CDAC to more accurately predict dialogue
acts in real users' conversations, suggesting a promising direction for future
improvements.

本文提出了一种名为 CDAC 的深度学习算法，通过在 Switchboard 人际对话数据集上进行预训练并在人机对话数据集上微调，实现了上下文对话行为分类，并表明采用小样本人机对话数据微调 CDAC 模型可以更精确地预测真实用户的对话行为，为未来的改进提供了有希望的方向。

面向开放域对话代理的上下文对话行为分类

Contextual Dialogue Act Classification for Open-Domain Conversational  Agents

Dialog act prediction is an essential language comprehension task for both
dialog system building and discourse analysis. Previous dialog act schemes,
such as SWBD-DAMSL, are designed for human-human conversations, in which
conversation partners have perfect language understanding ability. In this
paper, we design a dialog act annotation scheme, MIDAS (Machine Interaction
Dialog Act Scheme), targeted on open-domain human-machine conversations. MIDAS
is designed to assist machines which have limited ability to understand their
human partners. MIDAS has a hierarchical structure and supports multi-label
annotations. We collected and annotated a large open-domain human-machine
spoken conversation dataset (consists of 24K utterances). To show the
applicability of the scheme, we leverage transfer learning methods to train a
multi-label dialog act prediction model and reach an F1 score of 0.79.

本文设计了一个层次结构和支持多标签的对话行为注释方案 MIDAS，针对开放领域的人机对话。为了验证该方案的适用性，我们利用迁移学习方法训练多标签对话行为预测模型，并达到了 0.79 的 F1 分数。