Visual search is a ubiquitous challenge in natural vision, including daily
tasks such as finding a friend in a crowd or searching for a car in a parking
lot. Human rely heavily on relevant target features to perform goal-directed
visual search. Meanwhile, context is of critical importance for locating a
target object in complex scenes as it helps narrow down the search area and
makes the search process more efficient. However, few works have combined both
target and context information in visual search computational models. Here we
propose a zero-shot deep learning architecture, TCT (Target and Context-aware
Transformer), that modulates self attention in the Vision Transformer with
target and contextual relevant information to enable human-like zero-shot
visual search performance. Target modulation is computed as patch-wise local
relevance between the target and search images, whereas contextual modulation
is applied in a global fashion. We conduct visual search experiments on TCT and
other competitive visual search models on three natural scene datasets with
varying levels of difficulty. TCT demonstrates human-like performance in terms
of search efficiency and beats the SOTA models in challenging visual search
tasks. Importantly, TCT generalizes well across datasets with novel objects
without retraining or fine-tuning. Furthermore, we also introduce a new dataset
to benchmark models for invariant visual search under incongruent contexts. TCT
manages to search flexibly via target and context modulation, even under
incongruent contexts.

该研究提出了一种零样本深度学习架构 TCT（Target and Context-aware Transformer），结合目标和上下文信息进行图像搜索，表现出类人的搜索效率，在具有挑战性的视觉搜索任务中超越最先进的模型，并在具有不协调上下文的不变视觉搜索下展现出灵活的搜索方式。