Learning in simulation and transferring the learned policy to the real world
has the potential to enable generalist robots. The key challenge of this
approach is to address simulation-to-reality (sim-to-real) gaps. Previous
methods often require domain-specific knowledge a priori. We argue that a
straightforward way to obtain such knowledge is by asking humans to observe and
assist robot policy execution in the real world. The robots can then learn from
humans to close various sim-to-real gaps. We propose TRANSIC, a data-driven
approach to enable successful sim-to-real transfer based on a human-in-the-loop
framework. TRANSIC allows humans to augment simulation policies to overcome
various unmodeled sim-to-real gaps holistically through intervention and online
correction. Residual policies can be learned from human corrections and
integrated with simulation policies for autonomous execution. We show that our
approach can achieve successful sim-to-real transfer in complex and
contact-rich manipulation tasks such as furniture assembly. Through synergistic
integration of policies learned in simulation and from humans, TRANSIC is
effective as a holistic approach to addressing various, often coexisting
sim-to-real gaps. It displays attractive properties such as scaling with human
effort. Videos and code are available at this https URL

通过人机协作的数据驱动方法，TRANSIC 提出了一种能够成功实现从模拟环境到真实世界转换的综合性方法，通过引入人类观察和干预机器人在真实世界中的执行来缩小不同的模拟与现实之间的差距。该方法通过整合从模拟和人类中学习到的策略，在复杂和接触丰富的操作任务中实现了成功的模拟到现实的转换。

通过在线纠正学习进行模拟到现实策略迁移

TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction

When learning policies for robot control, the required real-world data is
typically prohibitively expensive to acquire, so learning in simulation is a
popular strategy. Unfortunately, such polices are often not transferable to the
real world due to a mismatch between the simulation and reality, called
'reality gap'. Domain randomization methods tackle this problem by randomizing
the physics simulator (source domain) during training according to a
distribution over domain parameters in order to obtain more robust policies
that are able to overcome the reality gap. Most domain randomization approaches
sample the domain parameters from a fixed distribution. This solution is
suboptimal in the context of sim-to-real transferability, since it yields
policies that have been trained without explicitly optimizing for the reward on
the real system (target domain). Additionally, a fixed distribution assumes
there is prior knowledge about the uncertainty over the domain parameters. In
this paper, we propose Bayesian Domain Randomization (BayRn), a black-box
sim-to-real algorithm that solves tasks efficiently by adapting the domain
parameter distribution during learning given sparse data from the real-world
target domain. BayRn uses Bayesian optimization to search the space of source
domain distribution parameters such that this leads to a policy which maximizes
the real-word objective, allowing for adaptive distributions during policy
optimization. We experimentally validate the proposed approach in sim-to-sim as
well as in sim-to-real experiments, comparing against three baseline methods on
two robotic tasks. Our results show that BayRn is able to perform sim-to-real
transfer, while significantly reducing the required prior knowledge.

本文提出了一种 Bayesian Domain Randomization（BayRn）算法，通过采样真实世界目标域中的稀疏数据来适应性地调整源域分布参数，解决在机器人控制中遇到的模拟培训与真实世界之间存在的不匹配问题，实现了在模拟和真实环境下的任务优化。