Token dropping is a recently-proposed strategy to speed up the pretraining of
masked language models, such as BERT, by skipping the computation of a subset
of the input tokens at several middle layers. It can effectively reduce the
training time without degrading much performance on downstream tasks. However,
we empirically find that token dropping is prone to a semantic loss problem and
falls short in handling semantic-intense tasks. Motivated by this, we propose a
simple yet effective semantic-consistent learning method (ScTD) to improve the
token dropping. ScTD aims to encourage the model to learn how to preserve the
semantic information in the representation space. Extensive experiments on 12
tasks show that, with the help of our ScTD, token dropping can achieve
consistent and significant performance gains across all task types and model
sizes. More encouragingly, ScTD saves up to 57% of pretraining time and brings
up to +1.56% average improvement over the vanilla token dropping.

通过提出语义一致性学习方法（ScTD）来优化 token dropping，从而在保持语义信息的同时显著提高性能，尤其在处理语义密集型任务时，ScTD 可以更好地处理语义丢失的问题。