Real-world data often have an open long-tailed distribution, and building a
unified QA model supporting various tasks is vital for practical QA
applications. However, it is non-trivial to extend previous QA approaches since
they either require access to seen tasks of adequate samples or do not
explicitly model samples from unseen tasks. In this paper, we define Open
Long-Tailed QA (OLTQA) as learning from long-tailed distributed data and
optimizing performance over seen and unseen QA tasks. We propose an OLTQA model
that encourages knowledge sharing between head, tail and unseen tasks, and
explicitly mines knowledge from a large pre-trained language model (LM).
Specifically, we organize our model through a pool of fine-grained components
and dynamically combine these components for an input to facilitate knowledge
sharing. A retrieve-then-rerank frame is further introduced to select
in-context examples, which guild the LM to generate text that express knowledge
for QA tasks. Moreover, a two-stage training approach is introduced to
pre-train the framework by knowledge distillation (KD) from the LM and then
jointly train the frame and a QA model through an adaptive mutual KD method. On
a large-scale OLTQA dataset we curate from 43 existing QA datasets, our model
consistently outperforms the state-of-the-art. We release the code and data at
https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/oltqa.

本文定义了 OLTQA 模型，它通过在预训练语言模型中挖掘知识和引入 retrieve-then-rerank 框架来支持各种 QA 任务，并提出了适应性的相互知识蒸馏方法联合训练框架和 QA 模型，在 43 个现有 QA 数据集上更好地执行。