Motivated by the Parameter-Efficient Fine-Tuning (PEFT) in large language
models, we propose LoRAT, a method that unveils the power of larger Vision
Transformers (ViT) for tracking within laboratory-level resources. The essence
of our work lies in adapting LoRA, a technique that fine-tunes a small subset
of model parameters without adding inference latency, to the domain of visual
tracking. However, unique challenges and potential domain gaps make this
transfer not as easy as the first intuition. Firstly, a transformer-based
tracker constructs unshared position embedding for template and search image.
This poses a challenge for the transfer of LoRA, usually requiring consistency
in the design when applied to the pre-trained backbone, to downstream tasks.
Secondly, the inductive bias inherent in convolutional heads diminishes the
effectiveness of parameter-efficient fine-tuning in tracking models. To
overcome these limitations, we first decouple the position embeddings in
transformer-based trackers into shared spatial ones and independent type ones.
The shared embeddings, which describe the absolute coordinates of
multi-resolution images (namely, the template and search images), are inherited
from the pre-trained backbones. In contrast, the independent embeddings
indicate the sources of each token and are learned from scratch. Furthermore,
we design an anchor-free head solely based on a multilayer perceptron (MLP) to
adapt PETR, enabling better performance with less computational overhead. With
our design, 1) it becomes practical to train trackers with the ViT-g backbone
on GPUs with only memory of 25.8GB (batch size of 16); 2) we reduce the
training time of the L-224 variant from 35.0 to 10.8 GPU hours; 3) we improve
the LaSOT SUC score from 0.703 to 0.743 with the L-224 variant; 4) we fast the
inference speed of the L-224 variant from 52 to 119 FPS. Code and models will
be released.

通过 LoRAT 方法，将 Vision Transformers (ViT) 应用于实验室级资源内的视觉跟踪，通过解决独特的挑战和潜在域差异，包括将位置嵌入分解为共享的空间嵌入和独立的类型嵌入，并设计了仅基于多层感知器 (MLP) 的自由锚定头进行适应，以提高性能并降低计算开销。

跟踪与 LoRA: 更快的训练、更大的模型、更强的性能

Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance

Transformer-based trackers have achieved strong accuracy on the standard
benchmarks. However, their efficiency remains an obstacle to practical
deployment on both GPU and CPU platforms. In this paper, to overcome this
issue, we propose a fully transformer tracking framework, coined as
\emph{MixFormerV2}, without any dense convolutional operation and complex score
prediction module. Our key design is to introduce four special prediction
tokens and concatenate them with the tokens from target template and search
areas. Then, we apply the unified transformer backbone on these mixed token
sequence. These prediction tokens are able to capture the complex correlation
between target template and search area via mixed attentions. Based on them, we
can easily predict the tracking box and estimate its confidence score through
simple MLP heads. To further improve the efficiency of MixFormerV2, we present
a new distillation-based model reduction paradigm, including dense-to-sparse
distillation and deep-to-shallow distillation. The former one aims to transfer
knowledge from the dense-head based MixViT to our fully transformer tracker,
while the latter one is used to prune some layers of the backbone. We
instantiate two types of MixForemrV2, where the MixFormerV2-B achieves an AUC
of 70.6\% on LaSOT and an AUC of 57.4\% on TNL2k with a high GPU speed of 165
FPS, and the MixFormerV2-S surpasses FEAR-L by 2.7\% AUC on LaSOT with a
real-time CPU speed.

本文提出了一种基于全 Transformer 结构的跟踪框架 MixFormerV2，通过引入四个特殊的预测 tokens，结合目标模板和搜索区域的 tokens，采用 Transformer 骨干网络进行跟踪预测，进而通过简单的 MLP heads 预测跟踪框和评估置信度得分，并提出了一种新的蒸馏模型的减少算法，包括密集到稀疏和深层到浅层的蒸馏，以进一步提高跟踪器性能。

MixFormerV2：高效全 Transformer 跟踪

MixFormerV2: Efficient Fully Transformer Tracking

Temporal modeling of objects is a key challenge in multiple object tracking
(MOT). Existing methods track by associating detections through motion-based
and appearance-based similarity heuristics. The post-processing nature of
association prevents end-to-end exploitation of temporal variations in video
sequence. In this paper, we propose MOTR, which extends DETR and introduces
track query to model the tracked instances in the entire video. Track query is
transferred and updated frame-by-frame to perform iterative prediction over
time. We propose tracklet-aware label assignment to train track queries and
newborn object queries. We further propose temporal aggregation network and
collective average loss to enhance temporal relation modeling. Experimental
results on DanceTrack show that MOTR significantly outperforms state-of-the-art
method, ByteTrack by 6.5% on HOTA metric. On MOT17, MOTR outperforms our
concurrent works, TrackFormer and TransTrack, on association performance. MOTR
can serve as a stronger baseline for future research on temporal modeling and
Transformer-based trackers. Code is available at
this https URL

本文介绍了一种名为 MOTR 的方法，它使用 track query 来建模整个视频中的跟踪实例，并引入了一种新的时域关系建模方法。实验结果表明，MOTR 在 HOTA 指标上比现有技术 ByteTrack 提高了 6.5%，在 MOT17 测试中，MOTR 的关联性能也胜过了 TrackFormer 和 TransTrack。