In this paper, we address the problem of using visuo-tactile feedback for
6-DoF localization and 3D reconstruction of unknown in-hand objects. We propose
FingerSLAM, a closed-loop factor graph-based pose estimator that combines local
tactile sensing at finger-tip and global vision sensing from a wrist-mount
camera. FingerSLAM is constructed with two constituent pose estimators: a
multi-pass refined tactile-based pose estimator that captures movements from
detailed local textures, and a single-pass vision-based pose estimator that
predicts from a global view of the object. We also design a loop closure
mechanism that actively matches current vision and tactile images to previously
stored key-frames to reduce accumulated error. FingerSLAM incorporates the two
sensing modalities of tactile and vision, as well as the loop closure mechanism
with a factor graph-based optimization framework. Such a framework produces an
optimized pose estimation solution that is more accurate than the standalone
estimators. The estimated poses are then used to reconstruct the shape of the
unknown object incrementally by stitching the local point clouds recovered from
tactile images. We train our system on real-world data collected with 20
objects. We demonstrate reliable visuo-tactile pose estimation and shape
reconstruction through quantitative and qualitative real-world evaluations on 6
objects that are unseen during training.

本文提出了 FingerSLAM，一种结合手指触觉传感和腕部摄像头视觉感知的封闭环路因子图姿态估计器，用于关于手中未知物体的 6DoF 定位和 3D 重建，通过该方法可以获得更加准确的估计，并示范了在实际情况下的定量和定性评估。