Robotic fabric manipulation has applications in home robotics, textiles,
senior care and surgery. Existing fabric manipulation techniques, however, are
designed for specific tasks, making it difficult to generalize across different
but related tasks. We build upon the Visual Foresight framework to learn fabric
dynamics that can be efficiently reused to accomplish different sequential
fabric manipulation tasks with a single goal-conditioned policy. We extend our
earlier work on VisuoSpatial Foresight (VSF), which learns visual dynamics on
domain randomized RGB images and depth maps simultaneously and completely in
simulation. In this earlier work, we evaluated VSF on multi-step fabric
smoothing and folding tasks against 5 baseline methods in simulation and on the
da Vinci Research Kit (dVRK) surgical robot without any demonstrations at train
or test time. A key finding was that depth sensing significantly improves
performance: RGBD data yields an 80% improvement in fabric folding success rate
in simulation over pure RGB data. In this work, we vary 4 components of VSF,
including data generation, visual dynamics model, cost function, and
optimization procedure. Results suggest that training visual dynamics models
using longer, corner-based actions can improve the efficiency of fabric folding
by 76% and enable a physical sequential fabric folding task that VSF could not
previously perform with 90% reliability. Code, data, videos, and supplementary
material are available at this https URL

该研究使用视觉预测框架来学习能在不同任务中重复使用的织物动力学，使用这种方法可以在织物平整和折叠任务中提高效率，并在物理上完成连续织物折叠任务。

物理序列织物操作的视觉空间预测

VisuoSpatial Foresight for Physical Sequential Fabric Manipulation

Robot learning has emerged as a promising tool for taming the complexity and
diversity of the real world. Methods based on high-capacity models, such as
deep networks, hold the promise of providing effective generalization to a wide
range of open-world environments. However, these same methods typically require
large amounts of diverse training data to generalize effectively. In contrast,
most robotic learning experiments are small-scale, single-domain, and
single-robot. This leads to a frequent tension in robotic learning: how can we
learn generalizable robotic controllers without having to collect impractically
large amounts of data for each separate experiment? In this paper, we propose
RoboNet, an open database for sharing robotic experience, which provides an
initial pool of 15 million video frames, from 7 different robot platforms, and
study how it can be used to learn generalizable models for vision-based robotic
manipulation. We combine the dataset with two different learning algorithms:
visual foresight, which uses forward video prediction models, and supervised
inverse models. Our experiments test the learned algorithms' ability to work
across new objects, new tasks, new scenes, new camera viewpoints, new grippers,
or even entirely new robots. In our final experiment, we find that by
pre-training on RoboNet and fine-tuning on data from a held-out Franka or Kuka
robot, we can exceed the performance of a robot-specific training approach that
uses 4x-20x more data. For videos and data, see the project webpage:
this https URL

本文提出 RoboNet，一个用于共享机器人经验的开放数据库，结合视觉先见性和监督逆模型两种学习算法，用于视觉引导的机器人操作的泛化模型学习，其预训练和微调性能超过使用 4 倍至 20 倍更多数据的机器人特定培训方法。