Existing pre-training methods for extractive Question Answering (QA) generate
cloze-like queries different from natural questions in syntax structure, which
could overfit pre-trained models to simple keyword matching. In order to
address this problem, we propose a novel Momentum Contrastive pRe-training fOr
queStion anSwering (MCROSS) method for extractive QA. Specifically, MCROSS
introduces a momentum contrastive learning framework to align the answer
probability between cloze-like and natural query-passage sample pairs. Hence,
the pre-trained models can better transfer the knowledge learned in cloze-like
samples to answering natural questions. Experimental results on three
benchmarking QA datasets show that our method achieves noticeable improvement
compared with all baselines in both supervised and zero-shot scenarios.

本研究提出了一种用于提取式问答系统的动量对比预训练方法（MCROSS），以解决现有预训练方法中存在的由于关键词匹配而导致过度拟合问题，该方法采用动量对比学习框架来对齐填空式和自然问题 - 文章样本对之间的答案概率，从而使预训练模型能够更好地将填空式中学到的知识应用于回答自然问题。实验结果表明，与所有基线模型相比，我们的方法在有监督和零 - shot 情况下均取得了显着的改进。