We introduce Bongard-OpenWorld, a new benchmark for evaluating real-world
few-shot reasoning for machine vision. It originates from the classical Bongard
Problems (BPs): Given two sets of images (positive and negative), the model
needs to identify the set that query images belong to by inducing the visual
concepts, which is exclusively depicted by images from the positive set. Our
benchmark inherits the few-shot concept induction of the original BPs while
adding the two novel layers of challenge: 1) open-world free-form concepts, as
the visual concepts in Bongard-OpenWorld are unique compositions of terms from
an open vocabulary, ranging from object categories to abstract visual
attributes and commonsense factual knowledge; 2) real-world images, as opposed
to the synthetic diagrams used by many counterparts. In our exploration,
Bongard-OpenWorld already imposes a significant challenge to current few-shot
reasoning algorithms. We further investigate to which extent the recently
introduced Large Language Models (LLMs) and Vision-Language Models (VLMs) can
solve our task, by directly probing VLMs, and combining VLMs and LLMs in an
interactive reasoning scheme. We even designed a neuro-symbolic reasoning
approach that reconciles LLMs & VLMs with logical reasoning to emulate the
human problem-solving process for Bongard Problems. However, none of these
approaches manage to close the human-machine gap, as the best learner achieves
64% accuracy while human participants easily reach 91%. We hope
Bongard-OpenWorld can help us better understand the limitations of current
visual intelligence and facilitate future research on visual agents with
stronger few-shot visual reasoning capabilities.

我们引入了 Bongard-OpenWorld，这是一个用于评估机器视觉领域的真实世界少样本推理的新基准。它继承了经典的 Bongard 问题的少样本概念归纳，并添加了两个新的挑战层：一是开放世界的自由形式概念，二是真实世界的图像。然而，目前的学习算法仍无法达到人类的准确度，我们希望 Bongard-OpenWorld 能帮助我们更好地理解当前视觉智能的局限性，并促进未来对具有较强少样本视觉推理能力的视觉智能体的研究。