Recently, spoken dialogue systems have been widely deployed in a variety of
applications, serving a huge number of end-users. A common issue is that the
errors resulting from noisy utterances, semantic misunderstandings, or lack of
knowledge make it hard for a real system to respond properly, possibly leading
to an unsatisfactory user experience. To avoid such a case, we consider a
proactive interaction mechanism where the system predicts the user satisfaction
with the candidate response before giving it to the user. If the user is not
likely to be satisfied according to the prediction, the system will ask the
user a suitable question to determine the real intent of the user instead of
providing the response directly. With such an interaction with the user, the
system can give a better response to the user. Previous models that predict the
user satisfaction are not applicable to DuerOS which is a large-scale
commercial dialogue system. They are based on hand-crafted features and thus
can hardly learn the complex patterns lying behind millions of conversations
and temporal dependency in multiple turns of the conversation. Moreover, they
are trained and evaluated on the benchmark datasets with adequate labels, which
are expensive to obtain in a commercial dialogue system. To face these
challenges, we propose a pipeline to predict the user satisfaction to help
DuerOS decide whether to ask for clarification in each turn. Specifically, we
propose to first generate a large number of weak labels and then train a
transformer-based model to predict the user satisfaction with these weak
labels. Empirically, we deploy and evaluate our model on DuerOS, and observe a
19% relative improvement on the accuracy of user satisfaction prediction and
2.3% relative improvement on user experience.

通过 proactive interaction mechanism 预测并提升用户满意度，本文提出了在 DuerOS 中使用基于 transformer 的模型进行大规模的用户满意度预测，能够有效提高用户体验。