Hand-eye calibration involves estimating the transformation between the camera and the robot. Traditional methods rely on fiducial markers, involving much manual labor and careful setup. Recent advancements in deep learning offer markerless techniques, but they present challenges, including the need for retraining networks for each robot, the requirement of accurate mesh models for data generation, and the need to address the sim-to-real gap. In this letter, we propose Kalib, an automatic and universal markerless hand-eye calibration pipeline that leverages the generalizability of visual foundation models to eliminate these barriers. In each calibration process, Kalib uses keypoint tracking and proprioceptive sensors to estimate the transformation between a robot's coordinate space and its corresponding points in camera space. Our method does not require training new networks or access to mesh models. Through evaluations in simulation environments and the real-world dataset DROID, Kalib demonstrates superior accuracy compared to recent baseline methods. This approach provides an effective and flexible calibration process for various robot systems by simplifying setup and removing dependency on precise physical markers.

本研究解决了传统手眼标定方法依赖于物理标记所带来的手动劳动和设置繁琐等问题。提出的Kalib方法通过利用视觉基础模型的通用性，实现自动化的无标记手眼标定过程，避免了对新网络训练和网格模型的需求。实验结果表明，Kalib在精度上优于现有基线方法，为各种机器人系统提供了一种灵活有效的标定流程。

Kalib: 无标记手眼标定与关键点跟踪