Object tracking is a fundamental task in computer vision with broad practical
applications across various domains, including traffic monitoring, robotics,
and autonomous vehicle tracking. In this project, we aim to develop a
sophisticated aerial vehicle system known as Track Anything Raptor (TAR),
designed to detect, segment, and track objects of interest based on
user-provided multimodal queries, such as text, images, and clicks. TAR
utilizes cutting-edge pre-trained models like DINO, CLIP, and SAM to estimate
the relative pose of the queried object. The tracking problem is approached as
a Visual Servoing task, enabling the UAV to consistently focus on the object
through advanced motion planning and control algorithms. We showcase how the
integration of these foundational models with a custom high-level control
algorithm results in a highly stable and precise tracking system deployed on a
custom-built PX4 Autopilot-enabled Voxl2 M500 drone. To validate the tracking
algorithm's performance, we compare it against Vicon-based ground truth.
Additionally, we evaluate the reliability of the foundational models in aiding
tracking in scenarios involving occlusions. Finally, we test and validate the
model's ability to work seamlessly with multiple modalities, such as click,
bounding box, and image templates.

开发了一种名为 Track Anything Raptor (TAR) 的先进空中飞行器系统，利用多模态请求 (如文本、图像和点击) 检测、分割和跟踪用户感兴趣的对象，并使用 DINO、CLIP 和 SAM 等先进模型估计查询对象的相对姿态，通过高级运动规划和控制算法，将跟踪问题解决为视觉伺服任务，在自定义高级控制算法与 PX4 自动驾驶仪及 Voxl2 M500 无人机的集成下展示了高度稳定和精准的跟踪系统，并通过与 Vicon 基准的比较验证了跟踪算法的性能，并评估了先进模型在涉及遮挡场景下的跟踪可靠性，最后测试和验证了模型在各种模态下的无缝工作能力，包括点击、包围框和图像模板。