Pre-trained Language Models (PTLMs) have been shown to perform well on
natural language tasks. Many prior works have leveraged structured commonsense
present in the form of entities linked through labeled relations in Knowledge
Graphs (KGs) to assist PTLMs. Retrieval approaches use KG as a separate static
module which limits coverage since KGs contain finite knowledge. Generative
methods train PTLMs on KG triples to improve the scale at which knowledge can
be obtained. However, training on symbolic KG entities limits their
applicability in tasks involving natural language text where they ignore
overall context. To mitigate this, we propose a CommonSense Contextualizer
(CoSe-Co) conditioned on sentences as input to make it generically usable in
tasks for generating knowledge relevant to the overall context of input text.
To train CoSe-Co, we propose a novel dataset comprising of sentence and
commonsense knowledge pairs. The knowledge inferred by CoSe-Co is diverse and
contain novel entities not present in the underlying KG. We augment generated
knowledge in Multi-Choice QA and Open-ended CommonSense Reasoning tasks leading
to improvements over current best methods on CSQA, ARC, QASC and OBQA datasets.
We also demonstrate its applicability in improving performance of a baseline
model for paraphrase generation task.

提出了一种基于符号知识图谱的预训练语言模型生成任务相关信息的 CommonSense Contextualizer (CoSe-Co) 模型，该模型结合句子作为输入文本的上下文信息，以构建一种新的数据集用于训练 CoSe-Co 模型，并在多项选择 QA 以及常识推理任务中展现出显著提高的性能

CoSe-Co: 文本条件生成通用常识上下文

CoSe-Co: Text Conditioned Generative CommonSense Contextualizer

Machine Reading Comprehension (MRC) for question answering (QA), which aims
to answer a question given the relevant context passages, is an important way
to test the ability of intelligence systems to understand human language.
Multiple-Choice QA (MCQA) is one of the most difficult tasks in MRC because it
often requires more advanced reading comprehension skills such as logical
reasoning, summarization, and arithmetic operations, compared to the extractive
counterpart where answers are usually spans of text within given passages.
Moreover, most existing MCQA datasets are small in size, making the learning
task even harder. We introduce MMM, a Multi-stage Multi-task learning framework
for Multi-choice reading comprehension. Our method involves two sequential
stages: coarse-tuning stage using out-of-domain datasets and multi-task
learning stage using a larger in-domain dataset to help model generalize better
with limited data. Furthermore, we propose a novel multi-step attention network
(MAN) as the top-level classifier for this task. We demonstrate MMM
significantly advances the state-of-the-art on four representative MCQA
datasets.

本研究提出了一种名为 MMM 的基于多阶段多任务学习框架的多选阅读理解方法，其中采用了一种名为多步注意力网络 (MAN) 的新型分类器。实验证明该方法在四个代表性 MCQA 数据集上显著提高了最新技术水平。