The field of robotics has made significant advances towards generalist robot
manipulation policies. However, real-world evaluation of such policies is not
scalable and faces reproducibility challenges, which are likely to worsen as
policies broaden the spectrum of tasks they can perform. We identify control
and visual disparities between real and simulated environments as key
challenges for reliable simulated evaluation and propose approaches for
mitigating these gaps without needing to craft full-fidelity digital twins of
real-world environments. We then employ these approaches to create SIMPLER, a
collection of simulated environments for manipulation policy evaluation on
common real robot setups. Through paired sim-and-real evaluations of
manipulation policies, we demonstrate strong correlation between policy
performance in SIMPLER environments and in the real world. Additionally, we
find that SIMPLER evaluations accurately reflect real-world policy behavior
modes such as sensitivity to various distribution shifts. We open-source all
SIMPLER environments along with our workflow for creating new environments at
this https URL to facilitate research on general-purpose
manipulation policies and simulated evaluation frameworks.

机器人学中的评估与模拟环境之间的控制和视觉差异是可靠的模拟评估的关键挑战，在创建适用于常见真实机器人设置的 SIMPLER 模拟环境的基础上，我们证明了在这些环境中的政策表现与真实世界中的表现之间的强相关性，同时准确反映了真实世界的政策行为模式，通过我们的工作流以及开源的 SIMPLER 环境，促进了通用操作策略和模拟评估框架的研究。