To realize effective large-scale, real-world robotic applications, we must
evaluate how well our robot policies adapt to changes in environmental
conditions. Unfortunately, a majority of studies evaluate robot performance in
environments closely resembling or even identical to the training setup. We
present THE COLOSSEUM, a novel simulation benchmark, with 20 diverse
manipulation tasks, that enables systematical evaluation of models across 12
axes of environmental perturbations. These perturbations include changes in
color, texture, and size of objects, table-tops, and backgrounds; we also vary
lighting, distractors, and camera pose. Using THE COLOSSEUM, we compare 4
state-of-the-art manipulation models to reveal that their success rate degrades
between 30-50% across these perturbation factors. When multiple perturbations
are applied in unison, the success rate degrades $\geq$75%. We identify that
changing the number of distractor objects, target object color, or lighting
conditions are the perturbations that reduce model performance the most. To
verify the ecological validity of our results, we show that our results in
simulation are correlated ($\bar{R}^2 = 0.614$) to similar perturbations in
real-world experiments. We open source code for others to use THE COLOSSEUM,
and also release code to 3D print the objects used to replicate the real-world
perturbations. Ultimately, we hope that THE COLOSSEUM will serve as a benchmark
to identify modeling decisions that systematically improve generalization for
manipulation. See this https URL for more details.

为了实现有效的大规模、实际应用的机器人，我们必须评估我们的机器人策略对环境条件变化的适应能力。我们介绍了一个新的模拟基准测试 COLOSSEUM，它包括 20 个不同的操作任务，能够系统地评估模型在 12 个环境扰动轴上的性能。我们比较了 4 种最先进的操作模型，揭示了它们在这些环境扰动因素下的成功率下降了 30-50%。当多个扰动同时应用时，成功率下降≥75%。我们发现改变干扰对象的数量、目标对象的颜色或照明条件是最大程度降低模型性能的扰动因素。为了验证我们结果的生态效度，我们证明了我们在模拟中的结果与真实实验中的类似扰动相关（R^2 = 0.614）。我们公开了 COLOSSEUM 的使用代码，并发布了用于复制真实世界扰动的三维打印对象的代码。最终，我们希望 THE COLOSSEUM 能成为一个评估操作泛化能力的基准，以找到能系统地改进操作泛化能力的建模决策。

THE COLOSSEUM: 机器人操作泛化评估基准

THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic  Manipulation

Sketch recognition algorithms are engineered and evaluated using publicly
available datasets contributed by the sketch recognition community over the
years. While existing datasets contain sketches of a limited set of generic
objects, each new domain inevitably requires collecting new data for training
domain specific recognizers. This gives rise to two fundamental concerns:
First, will the data collection protocol yield ecologically valid data? Second,
will the amount of collected data suffice to train sufficiently accurate
classifiers? In this paper, we draw attention to these two concerns. We show
that the ecological validity of the data collection protocol and the ability to
accommodate small datasets are significant factors impacting recognizer
accuracy in realistic scenarios. More specifically, using sketch-based gaming
as a use case, we show that deep learning methods, as well as more traditional
methods, suffer significantly from dataset shift. Furthermore, we demonstrate
that in realistic scenarios where data is scarce and expensive, standard
measures taken for adapting deep learners to small datasets fall short of
comparing favorably with alternatives. Although transfer learning, and
extensive data augmentation help deep learners, they still perform
significantly worse compared to standard setups (e.g., SVMs and GBMs with
standard feature representations). We pose learning from small datasets as a
key problem for the deep sketch recognition field, one which has been ignored
in the bulk of the existing literature.

本文研究草图识别算法，并关注数据收集协议的生态有效性以及适应小数据集的方法对算法的影响。通过以基于草图的游戏为例子，研究表明深度学习方法和 SVMs 以及 GBMs 相比，在数据不足且昂贵的情况下，随着数据集的增加，标准的方法表现效果更佳。在研究中，我们强调将小数据集的学习视为深度草图识别领域的一个重要问题，并针对这一问题提出了不同的解决方案。

训练新领域的草图识别器

On Training Sketch Recognizers for New Domains

Language User Interfaces (LUIs) could improve human-machine interaction for a
wide variety of tasks, such as playing music, getting insights from databases,
or instructing domestic robots. In contrast to traditional hand-crafted
approaches, recent work attempts to build LUIs in a data-driven way using
modern deep learning methods. To satisfy the data needs of such learning
algorithms, researchers have constructed benchmarks that emphasize the quantity
of collected data at the cost of its naturalness and relevance to real-world
LUI use cases. As a consequence, research findings on such benchmarks might not
be relevant for developing practical LUIs. The goal of this paper is to
bootstrap the discussion around this issue, which we refer to as the
benchmarks' low ecological validity. To this end, we describe what we deem an
ideal methodology for machine learning research on LUIs and categorize five
common ways in which recent benchmarks deviate from it. We give concrete
examples of the five kinds of deviations and their consequences. Lastly, we
offer a number of recommendations as to how to increase the ecological validity
of machine learning research on LUIs.

使用近代深度学习方法构建 Data-driven 的语言用户界面（LUIs），作者讨论了生成的数据来源对于实际应用的有效性的影响，指出当前一些基准测试对真实应用存在差距，提出了提高 LUIs 机器学习研究的生态效度的建议。