This paper presents the participation of NetEase Game AI Lab team for the
ClariQ challenge at Search-oriented Conversational AI (SCAI) EMNLP workshop in
2020. The challenge asks for a complete conversational information retrieval
system that can understanding and generating clarification questions. We
propose a clarifying question selection system which consists of response
understanding, candidate question recalling and clarifying question ranking. We
fine-tune a RoBERTa model to understand user's responses and use an enhanced
BM25 model to recall the candidate questions. In clarifying question ranking
stage, we reconstruct the training dataset and propose two models based on
ELECTRA. Finally we ensemble the models by summing up their output
probabilities and choose the question with the highest probability as the
clarification question. Experiments show that our ensemble ranking model
outperforms in the document relevance task and achieves the best recall@[20,30]
metrics in question relevance task. And in multi-turn conversation evaluation
in stage2, our system achieve the top score of all document relevance metrics.

该论文介绍了 NetEase 游戏 AI 实验室团队参加 2020 年 Search-oriented Conversational AI（SCAI）EMNLP 研讨会 ClariQ 挑战的情况。我们提出了一个澄清问题选择系统，它由回复理解、候选问题回忆和澄清问题排名组成。在实验中，我们使用 RoBERTa 模型来理解用户的回复，并使用 BM25 模型回忆候选问题。在澄清问题排名阶段，我们基于 ELECTRA 提出了两种模型，并将它们的输出概率相加，选择概率最高的问题作为澄清问题。实验结果表明，我们的模型在文档相关性任务中表现优异，且在问题相关性任务中达到了最佳的 recall@[20,30] 指标，而在第二阶段的多轮对话评估中，我们的系统获得了所有文档相关性度量的最高分数。