Salient Span Masking (SSM) has shown itself to be an effective strategy to
improve closed-book question answering performance. SSM extends general masked
language model pretraining by creating additional unsupervised training
sentences that mask a single entity or date span, thus oversampling factual
information. Despite the success of this paradigm, the span types and sampling
strategies are relatively arbitrary and not widely studied for other tasks.
Thus, we investigate SSM from the perspective of temporal tasks, where learning
a good representation of various temporal expressions is important. To that
end, we introduce Temporal Span Masking (TSM) intermediate training. First, we
find that SSM alone improves the downstream performance on three temporal tasks
by an avg. +5.8 points. Further, we are able to achieve additional improvements
(avg. +0.29 points) by adding the TSM task. These comprise the new best
reported results on the targeted tasks. Our analysis suggests that the
effectiveness of SSM stems from the sentences chosen in the training data
rather than the mask choice: sentences with entities frequently also contain
temporal expressions. Nonetheless, the additional targeted spans of TSM can
still improve performance, especially in a zero-shot context.

通过引入时间跨度遮蔽（TSM）中间训练，我们在已有的 Salient Span Masking（SSM）的基础上进行了研究，发现 SSM 单独对三项时间任务的下游性能有所提高，可以通过增加 TSM 任务进一步提高平均 0.29 分，并自主探究了 SSM 的知识来源与对模型性能的影响。

时间理解的显著跨度屏蔽

Salient Span Masking for Temporal Understanding

The ability of pretrained Transformers to remember factual knowledge is
essential but still limited for existing models. Inspired by existing work that
regards Feed-Forward Networks (FFNs) in Transformers as key-value memories, we
design a Neural Knowledge Bank (NKB) and a knowledge injection strategy to
introduce extra factual knowledge for pretrained Transformers. The NKB is in
the form of additional knowledgeable memory slots to the FFN and the
memory-like architecture makes it highly interpretable and flexible. When
injecting extra knowledge with the Salient Span Masking (SSM) pretraining
objective, we fix the original pretrained model and train only the NKB. This
training strategy makes sure the general language modeling ability of the
original pretrained model is not influenced. By mounting the NKB onto the T5
model, we verify its strong ability to store extra factual knowledge based on
three closed-book question answering datasets. Also, we prove that mounting the
NKB will not degrade the general language modeling ability of T5 through two
representative tasks, summarization and machine translation. Further, we
thoroughly analyze the interpretability of the NKB and reveal the meaning of
its keys and values in a human-readable way. Finally, we show the flexibility
of the NKB by directly modifying its value vectors to update the factual
knowledge stored in it.

本文提出一种神经知识库及知识注入策略，将额外的事实知识注入到预训练模型中，验证了其对模型性能的提升，同时保证了模型通用语言建模能力的不变性，并探讨了知识库的可解释性和灵活性。