In this paper, we present a multimodal mobile teleoperation system that
consists of a novel vision-based hand pose regression network (Transteleop) and
an IMU-based arm tracking method. Transteleop observes the human hand through a
low-cost depth camera and generates not only joint angles but also depth images
of paired robot hand poses through an image-to-image translation process. A
keypoint-based reconstruction loss explores the resemblance in appearance and
anatomy between human and robotic hands and enriches the local features of
reconstructed images. A wearable camera holder enables simultaneous hand-arm
control and facilitates the mobility of the whole teleoperation system. Network
evaluation results on a test dataset and a variety of complex manipulation
tasks that go beyond simple pick-and-place operations show the efficiency and
stability of our multimodal teleoperation system.

本文介绍了一个多模态移动远程操作系统，其由一种新颖的基于视觉的手部姿态回归网络（Transteleop）和一种基于 IMU 的臂部追踪方法组成，并使用低成本深度摄像头观察人手，并通过图像到图像的转换过程生成了配对机器人手部姿势的深度图像，同时具备同时手 - 臂控制的可穿戴相机支架，关键点重建损失探索了人类和机器人手之间在外观和解剖上的相似之处，并丰富了重建图像的局部特征。网络评价结果和各种复杂操作的操作都表明了我们的系统的效率和稳定性。