Tactile sensing is one of the modalities humans rely on heavily to perceive
the world. Working with vision, this modality refines local geometry structure,
measures deformation at the contact area, and indicates the hand-object contact
state.
With the availability of open-source tactile sensors such as DIGIT, research
on visual-tactile learning is becoming more accessible and reproducible.
Leveraging this tactile sensor, we propose a novel visual-tactile in-hand
object reconstruction framework \textbf{VTacO}, and extend it to
\textbf{VTacOH} for hand-object reconstruction. Since our method can support
both rigid and deformable object reconstruction, no existing benchmarks are
proper for the goal. We propose a simulation environment, VT-Sim, which
supports generating hand-object interaction for both rigid and deformable
objects. With VT-Sim, we generate a large-scale training dataset and evaluate
our method on it. Extensive experiments demonstrate that our proposed method
can outperform the previous baseline methods qualitatively and quantitatively.
Finally, we directly apply our model trained in simulation to various
real-world test cases, which display qualitative results.
Codes, models, simulation environment, and datasets are available at
https://sites.google.com/view/vtaco/.

本研究提出了一种基于 DIGIT 触觉传感器的视触对象重建框架 VTacO，并将其扩展到手 - 物体重建的 VTacOH，并通过 VT-Sim 模拟环境生成了大规模的训练数据集进行评估，展示了其在刚性和不可变形物体重建上具有优异的表现。