Motivated by the Parameter-Efficient Fine-Tuning (PEFT) in large language
models, we propose LoRAT, a method that unveils the power of larger Vision
Transformers (ViT) for tracking within laboratory-level resources. The essence
of our work lies in adapting LoRA, a technique that fine-tunes a small subset
of model parameters without adding inference latency, to the domain of visual
tracking. However, unique challenges and potential domain gaps make this
transfer not as easy as the first intuition. Firstly, a transformer-based
tracker constructs unshared position embedding for template and search image.
This poses a challenge for the transfer of LoRA, usually requiring consistency
in the design when applied to the pre-trained backbone, to downstream tasks.
Secondly, the inductive bias inherent in convolutional heads diminishes the
effectiveness of parameter-efficient fine-tuning in tracking models. To
overcome these limitations, we first decouple the position embeddings in
transformer-based trackers into shared spatial ones and independent type ones.
The shared embeddings, which describe the absolute coordinates of
multi-resolution images (namely, the template and search images), are inherited
from the pre-trained backbones. In contrast, the independent embeddings
indicate the sources of each token and are learned from scratch. Furthermore,
we design an anchor-free head solely based on a multilayer perceptron (MLP) to
adapt PETR, enabling better performance with less computational overhead. With
our design, 1) it becomes practical to train trackers with the ViT-g backbone
on GPUs with only memory of 25.8GB (batch size of 16); 2) we reduce the
training time of the L-224 variant from 35.0 to 10.8 GPU hours; 3) we improve
the LaSOT SUC score from 0.703 to 0.743 with the L-224 variant; 4) we fast the
inference speed of the L-224 variant from 52 to 119 FPS. Code and models will
be released.

通过 LoRAT 方法，将 Vision Transformers (ViT) 应用于实验室级资源内的视觉跟踪，通过解决独特的挑战和潜在域差异，包括将位置嵌入分解为共享的空间嵌入和独立的类型嵌入，并设计了仅基于多层感知器 (MLP) 的自由锚定头进行适应，以提高性能并降低计算开销。