Robot evaluations in language-guided, real world settings are time-consuming
and often sample only a small space of potential instructions across complex
scenes. In this work, we introduce contrast sets for robotics as an approach to
make small, but specific, perturbations to otherwise independent, identically
distributed (i.i.d.) test instances. We investigate the relationship between
experimenter effort to carry out an evaluation and the resulting estimated test
performance as well as the insights that can be drawn from performance on
perturbed instances. We use contrast sets to characterize policies at reduced
experimenter effort in both a simulated manipulation task and a physical robot
vision-and-language navigation task. We encourage the use of contrast set
evaluations as a more informative alternative to small scale, i.i.d.
demonstrations on physical robots, and as a scalable alternative to
industry-scale real world evaluations.

在语言引导的真实环境中，机器人评估需要耗费时间且通常只能对复杂场景中的潜在指令空间进行小规模采样。本研究提出了对机器人引入对比集的方法，通过对独立同分布（i.i.d.）的测试实例进行小、但具体的扰动。我们研究了评估工作的执行难度与估计测试性能之间的关系，以及在扰动实例上的性能对研究的启示。我们利用对比集来对模拟操纵任务和物理机器人的视觉与语言导航任务中的策略进行特征化，并鼓励使用对比集评估作为对物理机器人小规模 i.i.d. 演示的更具信息量的替代方案，以及对工业规模的真实环境评估的可扩展替代方案。