Training models that are robust to data domain shift has gained an increasing interest both in academia and industry. Question-Answering language models, being one of the typical problem in Natural Language Processing (NLP) research, has received much success with the advent of large transformer models. However, existing approaches mostly work under the assumption that data is drawn from same distribution during training and testing which is unrealistic and non-scalable in the wild. In this paper, we explore adversarial training approach towards learning domain-invariant features so that language models can generalize well to out-of-domain datasets. We also inspect various other ways to boost our model performance including data augmentation by paraphrasing sentences, conditioning end of answer span prediction on the start word, and carefully designed annealing function. Our initial results show that in combination with these methods, we are able to achieve $15.2\%$ improvement in EM score and $5.6\%$ boost in F1 score on out-of-domain validation dataset over the baseline. We also dissect our model outputs and visualize the model hidden-states by projecting them onto a lower-dimensional space, and discover that our specific adversarial training approach indeed encourages the model to learn domain invariant embedding and bring them closer in the multi-dimensional space.

探索对抗训练方法以学习领域不变特征，从而使语言模型可以对域外数据集进行良好的推广。我们还审查了各种其他方法来提高我们的模型性能，包括通过改写句子实现数据增强、将回答跨度预测的结尾词映射到开始词上以及精心设计的退火函数。我们的初步结果表明，在结合这些方法的情况下，与基线相比，在域外验证数据集上我们能够实现EM分数15.2%的提高和F1分数5.6%的提高。

QAGAN：学习领域不变的语言特征的对抗方法