Keypoint-based representation has proven advantageous in various visual and
robotic tasks. However, the existing 2D and 3D methods for detecting keypoints
mainly rely on geometric consistency to achieve spatial alignment, neglecting
temporal consistency. To address this issue, the Transporter method was
introduced for 2D data, which reconstructs the target frame from the source
frame to incorporate both spatial and temporal information. However, the direct
application of the Transporter to 3D point clouds is infeasible due to their
structural differences from 2D images. Thus, we propose the first 3D version of
the Transporter, which leverages hybrid 3D representation, cross attention, and
implicit reconstruction. We apply this new learning system on 3D articulated
objects and nonrigid animals (humans and rodents) and show that learned
keypoints are spatio-temporally consistent. Additionally, we propose a
closed-loop control strategy that utilizes the learned keypoints for 3D object
manipulation and demonstrate its superior performance. Codes are available at
this https URL

采用关键点为基础的表征在各种视觉和机器人任务中证明了优势。然而，现有的二维和三维关键点检测方法主要依赖几何一致性实现空间对齐，忽视了时间一致性。为了解决这个问题，介绍了用于二维数据的传输器方法，从源帧重构目标帧以包含空间和时间信息。然而，将传输器直接应用于三维点云由于其与二维图像的结构差异而不可行。因此，我们提出了第一个三维版本的传输器，它利用混合三维表示、交叉关注和隐式重构。我们将这个新的学习系统应用于三维关节对象和非刚性动物（人类和啮齿动物），并展示了学到的关键点在时空上的一致性。此外，我们提出了一种闭环控制策略，利用学到的关键点进行三维物体操作，并展示了其卓越性能。代码可在此 https URL 获取。