Recent studies report that many machine reading comprehension (MRC) models
can perform closely to or even better than humans on benchmark datasets.
However, existing works indicate that many MRC models may learn shortcuts to
outwit these benchmarks, but the performance is unsatisfactory in real-world
applications. In this work, we attempt to explore, instead of the expected
comprehension skills, why these models learn the shortcuts. Based on the
observation that a large portion of questions in current datasets have shortcut
solutions, we argue that larger proportion of shortcut questions in training
data make models rely on shortcut tricks excessively. To investigate this
hypothesis, we carefully design two synthetic datasets with annotations that
indicate whether a question can be answered using shortcut solutions. We
further propose two new methods to quantitatively analyze the learning
difficulty regarding shortcut and challenging questions, and revealing the
inherent learning mechanism behind the different performance between the two
kinds of questions. A thorough empirical analysis shows that MRC models tend to
learn shortcut questions earlier than challenging questions, and the high
proportions of shortcut questions in training sets hinder models from exploring
the sophisticated reasoning skills in the later stage of training.

本文研究了机器阅读理解 (MRC) 模型的学习过程，发现在训练数据中存在大量的提供快捷解决方案的问题，并且 MRC 模型倾向于优先学习快捷问题，而后期训练中高比例的快捷问题会阻碍模型深入探索复杂的推理技能。