The world around us is full of soft objects that we as humans learn to
perceive and deform with dexterous hand movements from a young age. In order
for a Robotic hand to be able to control soft objects, it needs to acquire
online state feedback of the deforming object. While RGB-D cameras can collect
occluded information at a rate of 30 Hz, the latter does not represent a
continuously trackable object surface. Hence, in this work, we developed a
method that can create deforming meshes of deforming point clouds at a speed of
above 50 Hz for different categories of objects. The reconstruction of meshes
from point clouds has been long studied in the field of Computer graphics under
3D reconstruction and 4D reconstruction, however both lack the speed and
generalizability needed for robotics applications. Our model is designed using
a point cloud auto-encoder and a Real-NVP architecture. The latter is a
continuous flow neural network with manifold-preservation properties. Our model
takes a template mesh which is the mesh of an object in its canonical state and
then deforms the template mesh to match a deformed point cloud of the object.
Our method can perform mesh reconstruction and tracking at a rate of 58 Hz for
deformations of six different ycb categories. An instance of a downstream
application can be the control algorithm for a robotic hand that requires
online feedback from the state of a manipulated object which would allow online
grasp adaptation in a closed-loop manner. Furthermore, the tracking capacity
that our method provides can help in the system identification of deforming
objects in a marker-free approach. In future work, we will extend our method to
more categories of objects and real world deforming point clouds

我们开发了一种能够以超过 50 赫兹的速度为不同类别的对象创建变形点云的变形网格的方法，该方法使用点云自动编码器和 Real-NVP 架构设计，可以在闭环方式下提供在线反馈并进行机器人应用的网格重建和跟踪。