Unsupervised domain adaptation (UDA) aims to transfer the knowledge learnt
from a labeled source domain to an unlabeled target domain. Previous work is
mainly built upon convolutional neural networks (CNNs) to learn
domain-invariant representations. With the recent exponential increase in
applying Vision Transformer (ViT) to vision tasks, the capability of ViT in
adapting cross-domain knowledge, however, remains unexplored in the literature.
To fill this gap, this paper first comprehensively investigates the
transferability of ViT on a variety of domain adaptation tasks. Surprisingly,
ViT demonstrates superior transferability over its CNNs-based counterparts with
a large margin, while the performance can be further improved by incorporating
adversarial adaptation. Notwithstanding, directly using CNNs-based adaptation
strategies fails to take the advantage of ViT's intrinsic merits (e.g.,
attention mechanism and sequential image representation) which play an
important role in knowledge transfer. To remedy this, we propose an unified
framework, namely Transferable Vision Transformer (TVT), to fully exploit the
transferability of ViT for domain adaptation. Specifically, we delicately
devise a novel and effective unit, which we term Transferability Adaption
Module (TAM). By injecting learned transferabilities into attention blocks, TAM
compels ViT focus on both transferable and discriminative features. Besides, we
leverage discriminative clustering to enhance feature diversity and separation
which are undermined during adversarial domain alignment. To verify its
versatility, we perform extensive studies of TVT on four benchmarks and the
experimental results demonstrate that TVT attains significant improvements
compared to existing state-of-the-art UDA methods.

本研究提出了一种名为 “可传递视觉 Transformer” 的统一框架，其中融合了迁移学习、注意力机制和聚类方法，以实现基于标签源域的无标签目标域的知识迁移。 实验证明，本文所提出的 TVT 方法优于现有的微调和迁移学习方法，通过注入可学习的传递性适应模块来强制 ViT 集中注意力在可转移和辨别性特征上，并借助判别性聚类来增强特征多样性和分离度。