Human capabilities in understanding visual relations are far superior to those of AI systems, especially for previously unseen objects. For example, while AI systems struggle to determine whether two such objects are visually the same or different, humans can do so with ease. Active vision theories postulate that the learning of visual relations is grounded in actions that we take to fixate objects and their parts by moving our eyes. In particular, the low-dimensional spatial information about the corresponding eye movements is hypothesized to facilitate the representation of relations between different image parts. Inspired by these theories, we develop a system equipped with a novel Glimpse-based Active Perception (GAP) that sequentially glimpses at the most salient regions of the input image and processes them at high resolution. Importantly, our system leverages the locations stemming from the glimpsing actions, along with the visual content around them, to represent relations between different parts of the image. The results suggest that the GAP is essential for extracting visual relations that go beyond the immediate visual content. Our approach reaches state-of-the-art performance on several visual reasoning tasks being more sample-efficient, and generalizing better to out-of-distribution visual inputs than prior models.

本研究解决了AI系统在理解视觉关系时面临的挑战，特别是对于未见过对象的处理。通过引入一种新颖的基于瞥见的主动感知系统，该方法利用瞥见动作生成的位置信息和周围视觉内容，成功提取超出直接视觉内容的视觉关系，最终在多项视觉推理任务上实现了先进的性能，展示了更好的样本效率和泛化能力。

注意间隙：基于瞥见的主动感知改善视觉推理的泛化能力和样本效率