Chatbots are expected to be knowledgeable across multiple domains, e.g. for daily chit-chat, exchange of information, and grounding in emotional situations. To effectively measure the quality of such conversational agents, a model-based automatic dialogue evaluation metric (ADEM) is expected to perform well across multiple domains. Despite significant progress, an ADEM that works well in one domain does not necessarily generalize to another. This calls for a dedicated network architecture for domain generalization. To tackle the multi-domain dialogue evaluation task, we propose a Panel of Experts (PoE), a multitask network that consists of a shared transformer encoder and a collection of lightweight adapters. The shared encoder captures the general knowledge of dialogues across domains, while each adapter specializes in one specific domain and serves as a domain expert. To validate the idea, we construct a high-quality multi-domain dialogue dataset leveraging data augmentation and pseudo-labeling. The PoE network is comprehensively assessed on 16 dialogue evaluation datasets spanning a wide range of dialogue domains. It achieves state-of-the-art performance in terms of mean Spearman correlation over all the evaluation datasets. It exhibits better zero-shot generalization than existing state-of-the-art ADEMs and the ability to easily adapt to new domains with few-shot transfer learning.

通过构建一个专门为领域通用而设计的Panel of Experts（PoE）网络，我们实现了一个能够在多个对话领域实现最佳表现的其次对话评估度量标准（ADEM）。该网络包含了共享编码器和多个适应器，每个适应器都专注于一项特定领域并充当其领域专家。该模型以平均斯皮尔曼相关性的方式取得了各项评估指标上的最新成果，同时表现出比现有ADEMs更好的零-shot泛化能力，以及容易适应于少量转移学习的新领域。

PoE：面向广义自动对话评估的专家小组