Explicit decomposition modeling, which involves breaking down complex tasks
into more straightforward and often more interpretable sub-tasks, has long been
a central theme in developing robust and interpretable NLU systems. However,
despite the many datasets and resources built as part of this effort, the
majority have small-scale annotations and limited scope, which is insufficient
to solve general decomposition tasks. In this paper, we look at large-scale
intermediate pre-training of decomposition-based transformers using distant
supervision from comparable texts, particularly large-scale parallel news. We
show that with such intermediate pre-training, developing robust
decomposition-based models for a diverse range of tasks becomes more feasible.
For example, on semantic parsing, our model, DecompT5, improves 20% to 30% on
two datasets, Overnight and TORQUE, over the baseline language model. We
further use DecompT5 to build a novel decomposition-based QA system named
DecompEntail, improving over state-of-the-art models, including GPT-3, on both
HotpotQA and StrategyQA by 8% and 4%, respectively.

本文研究大规模中介设备的预训练，在使用来自可比文本，特别是大规模平行新闻的远程监督的情况下将显式分解建模应用于自然语言理解系统。作者称使用这种中间预训练可以更容易地开发强大的显式分解模型。例如，该模型在语义分析方面取得了 20％至 30％的改进，并用于构建名为 DecompEntail 的新型显式分解问答系统。