Image retrieval with natural language feedback offers the promise of catalog
search based on fine-grained visual features that go beyond objects and binary
attributes, facilitating real-world applications such as e-commerce. Our
Modality-Agnostic Attention Fusion (MAAF) model combines image and text
features and outperforms existing approaches on two visual search with
modifying phrase datasets, Fashion IQ and CSS, and performs competitively on a
dataset with only single-word modifications, Fashion200k. We also introduce two
new challenging benchmarks adapted from Birds-to-Words and Spot-the-Diff, which
provide new settings with rich language inputs, and we show that our approach
without modification outperforms strong baselines. To better understand our
model, we conduct detailed ablations on Fashion IQ and provide visualizations
of the surprising phenomenon of words avoiding "attending" to the image region
they refer to.

本研究采用自然语言反馈的图像检索方法，结合图像与文本特征实现细粒度视觉搜索并提出了 MAAF 模型，在 Fashion IQ 和 CSS 数据集中比现有方法表现更优，同时在 Fashion200k 数据集中也取得竞争性表现。此外，我们还提出了两个适用于丰富语言输入的新挑战基准，并通过实验证明该方法在不修改时优于强基线。最后我们在 Fashion IQ 上进行了深入细致的分析和可视化，揭示了单词避免 “关注” 他们所指图像区域的惊人现象