The systematic evaluation and understanding of computer vision models under
varying conditions require large amounts of data with comprehensive and
customized labels, which real-world vision datasets rarely satisfy. While
current synthetic data generators offer a promising alternative, particularly
for embodied AI tasks, they often fall short for computer vision tasks due to
low asset and rendering quality, limited diversity, and unrealistic physical
properties. We introduce the BEHAVIOR Vision Suite (BVS), a set of tools and
assets to generate fully customized synthetic data for systematic evaluation of
computer vision models, based on the newly developed embodied AI benchmark,
BEHAVIOR-1K. BVS supports a large number of adjustable parameters at the scene
level (e.g., lighting, object placement), the object level (e.g., joint
configuration, attributes such as "filled" and "folded"), and the camera level
(e.g., field of view, focal length). Researchers can arbitrarily vary these
parameters during data generation to perform controlled experiments. We
showcase three example application scenarios: systematically evaluating the
robustness of models across different continuous axes of domain shift,
evaluating scene understanding models on the same set of images, and training
and evaluating simulation-to-real transfer for a novel vision task: unary and
binary state prediction. Project website:
this https URL

介绍了 BEHAVIOR Vision Suite（BVS），它是一套用于生成全面定制的合成数据以系统评估计算机视觉模型的工具和资源，并且支持在场景、对象和摄像机级别进行可调参数的任意变化，以执行可控实验。