This report proposes an improved method for the Tracking Any Point (TAP)
task, which tracks any physical surface through a video. Several existing
approaches have explored the TAP by considering the temporal relationships to
obtain smooth point motion trajectories, however, they still suffer from the
cumulative error caused by temporal prediction. To address this issue, we
propose a simple yet effective approach called TAP with confident static points
(TAPIR+), which focuses on rectifying the tracking of the static point in the
videos shot by a static camera. To clarify, our approach contains two key
components: (1) Multi-granularity Camera Motion Detection, which could identify
the video sequence by the static camera shot. (2) CMR-based point trajectory
prediction with one moving object segmentation approach to isolate the static
point from the moving object. Our approach ranked first in the final test with
a score of 0.46.

本文提出了一种改进的 TAP 方法，通过跟踪视频中的任何物理表面，旨在解决现有方法在处理累积误差问题时的不足，并介绍了 TAPIR + 方法，其中包含多粒度摄像机运动检测和基于 CMR 的点轨迹预测等关键组件。该方法在最终测试中得分 0.46，取得了第一名的成绩。

2023 ICCV 第一届感知测试挑战赛点追踪任务解决方案

Solution for Point Tracking Task of ICCV 1st Perception Test Challenge  2023

In this paper, we propose a simple and strong framework for Tracking Any
Point with TRansformers (TAPTR). Based on the observation that point tracking
bears a great resemblance to object detection and tracking, we borrow designs
from DETR-like algorithms to address the task of TAP. In the proposed
framework, in each video frame, each tracking point is represented as a point
query, which consists of a positional part and a content part. As in DETR, each
query (its position and content feature) is naturally updated layer by layer.
Its visibility is predicted by its updated content feature. Queries belonging
to the same tracking point can exchange information through self-attention
along the temporal dimension. As all such operations are well-designed in
DETR-like algorithms, the model is conceptually very simple. We also adopt some
useful designs such as cost volume from optical flow models and develop simple
designs to provide long temporal information while mitigating the feature
drifting issue. Our framework demonstrates strong performance with
state-of-the-art performance on various TAP datasets with faster inference
speed.

基于对点追踪与目标检测的相似性的观察，本文提出了一种简单而强大的使用 TRansformers 进行 Tracking Any Point（TAPTR）的框架。

TAPTR：使用 Transformers 作为检测器的任意点跟踪

TAPTR: Tracking Any Point with Transformers as Detection

We present a novel model for Tracking Any Point (TAP) that effectively tracks
any queried point on any physical surface throughout a video sequence. Our
approach employs two stages: (1) a matching stage, which independently locates
a suitable candidate point match for the query point on every other frame, and
(2) a refinement stage, which updates both the trajectory and query features
based on local correlations. The resulting model surpasses all baseline methods
by a significant margin on the TAP-Vid benchmark, as demonstrated by an
approximate 20% absolute average Jaccard (AJ) improvement on DAVIS. Our model
facilitates fast inference on long and high-resolution video sequences. On a
modern GPU, our implementation has the capacity to track points faster than
real-time. Visualizations, source code, and pretrained models can be found on
our project webpage.

本研究提出了一种名为 TAP 的新型模型，能在视频序列中高效地跟踪任何查询点，包含匹配阶段和细化阶段，且比基准方法表现更优，可快速适用于长且高分辨率的视频序列，具有实时的跟踪速度。

TAPIR: 使用逐帧初始化和时间细化技术追踪任意点

TAPIR: Tracking Any Point with per-frame Initialization and temporal  Refinement

We tackle the problem of Tracking Any Point (TAP) in videos, which
specifically aims at estimating persistent long-term trajectories of query
points in videos. Previous methods attempted to estimate these trajectories
independently to incorporate longer image sequences, therefore, ignoring the
potential benefits of incorporating spatial context features. We argue that
independent video point tracking also demands spatial context features. To this
end, we propose a novel framework Context-TAP, which effectively improves point
trajectory accuracy by aggregating spatial context features in videos.
Context-TAP contains two main modules: 1) a SOurse Feature Enhancement (SOFE)
module, and 2) a TArget Feature Aggregation (TAFA) module. Context-TAP
significantly improves PIPs all-sided, reducing 11.4% Average Trajectory Error
of Occluded Points (ATE-Occ) on CroHD and increasing 11.8% Average Percentage
of Correct Keypoint (A-PCK) on TAP-Vid-Kinectics. Demos are available at this
$\href{this https URL}{webpage}$.

该研究提出了一种新的视频点追踪框架 Context-TAP，通过聚合空间上下文特征，有效地提高了点轨迹的准确性。