Self-supervised learning (SSL), as a newly emerging unsupervised
representation learning paradigm, generally follows a two-stage learning
pipeline: 1) learning invariant and discriminative representations with
auto-annotation pretext(s), then 2) transferring the representations to assist
downstream task(s). Such two stages are usually implemented separately, making
the learned representation learned agnostic to the downstream tasks. Currently,
most works are devoted to exploring the first stage. Whereas, it is less
studied on how to learn downstream tasks with limited labeled data using the
already learned representations. Especially, it is crucial and challenging to
selectively utilize the complementary representations from diverse pretexts for
a downstream task. In this paper, we technically propose a novel solution by
leveraging the attention mechanism to adaptively squeeze suitable
representations for the tasks. Meanwhile, resorting to information theory, we
theoretically prove that gathering representation from diverse pretexts is more
effective than a single one. Extensive experiments validate that our scheme
significantly exceeds current popular pretext-matching based methods in
gathering knowledge and relieving negative transfer in downstream tasks.

本文提出一项新方案，即利用注意机制自适应地挑选适合任务的表示，同时在信息论的基础上理论证明从多个预训练任务聚集表示比从单一预训练任务获取表示更为有效，大量实验表明，该方案明显优于当前流行的预文本匹配方法，能有效地收集知识和减轻下游任务中的负转移。