Dialogue level quality estimation is vital for optimizing data driven
dialogue management. Current automated methods to estimate turn and dialogue
level user satisfaction employ hand-crafted features and rely on complex
annotation schemes, which reduce the generalizability of the trained models. We
propose a novel user satisfaction estimation approach which minimizes an
adaptive multi-task loss function in order to jointly predict turn-level
Response Quality labels provided by experts and explicit dialogue-level ratings
provided by end users. The proposed BiLSTM based deep neural net model
automatically weighs each turn's contribution towards the estimated
dialogue-level rating, implicitly encodes temporal dependencies, and removes
the need to hand-craft features.
On dialogues sampled from 28 Alexa domains, two dialogue systems and three
user groups, the joint dialogue-level satisfaction estimation model achieved up
to an absolute 27% (0.43->0.70) and 7% (0.63->0.70) improvement in linear
correlation performance over baseline deep neural net and benchmark Gradient
boosting regression models, respectively.

提出了一种新颖的用户满意度估计方法，它可通过 BiLSTM 深度神经网络模型共同预测专家提供的转化级别响应质量标签和最终用户提供的显式对话级别评分，并自动权衡每个操作对估计对话级别评分的贡献，隐式编码时间依赖关系，无需手工制作特征。在 28 个 Alexa 领域、两个对话系统和三个用户组的对话中，综合对话级满意度估计模型比基准深度神经网络和梯度提升回归模型在线性相关性能上实现了最高达 27%（0.43->0.70）和 7%（0.63->0.70）的提升。