Attribute-specific fashion retrieval (ASFR) is a challenging information
retrieval task, which has attracted increasing attention in recent years.
Different from traditional fashion retrieval which mainly focuses on optimizing
holistic similarity, the ASFR task concentrates on attribute-specific
similarity, resulting in more fine-grained and interpretable retrieval results.
As the attribute-specific similarity typically corresponds to the specific
subtle regions of images, we propose a Region-to-Patch Framework (RPF) that
consists of a region-aware branch and a patch-aware branch to extract
fine-grained attribute-related visual features for precise retrieval in a
coarse-to-fine manner. In particular, the region-aware branch is first to be
utilized to locate the potential regions related to the semantic of the given
attribute. Then, considering that the located region is coarse and still
contains the background visual contents, the patch-aware branch is proposed to
capture patch-wise attribute-related details from the previous amplified
region. Such a hybrid architecture strikes a proper balance between region
localization and feature extraction. Besides, different from previous works
that solely focus on discriminating the attribute-relevant foreground visual
features, we argue that the attribute-irrelevant background features are also
crucial for distinguishing the detailed visual contexts in a contrastive
manner. Therefore, a novel E-InfoNCE loss based on the foreground and
background representations is further proposed to improve the discrimination of
attribute-specific representation. Extensive experiments on three datasets
demonstrate the effectiveness of our proposed framework, and also show a decent
generalization of our RPF on out-of-domain fashion images. Our source code is
available at this https URL

本文提出了一种基于 Region-to-Patch 框架以及 E-InfoNCE loss 的属性特定时尚检索系统，通过提取细粒度视觉特征，将属性感知的相似度与以往的相似度方法相结合，从而在细节上提高检索结果的准确性。

从区域到补丁：面向属性的前景 - 背景对比学习用于细粒度时尚检索

From Region to Patch: Attribute-Aware Foreground-Background Contrastive  Learning for Fine-Grained Fashion Retrieval

This paper presents Contrastive Reconstruction, ConRec - a self-supervised
learning algorithm that obtains image representations by jointly optimizing a
contrastive and a self-reconstruction loss. We showcase that state-of-the-art
contrastive learning methods (e.g. SimCLR) have shortcomings to capture
fine-grained visual features in their representations. ConRec extends the
SimCLR framework by adding (1) a self-reconstruction task and (2) an attention
mechanism within the contrastive learning task. This is accomplished by
applying a simple encoder-decoder architecture with two heads. We show that
both extensions contribute towards an improved vector representation for images
with fine-grained visual features. Combining those concepts, ConRec outperforms
SimCLR and SimCLR with Attention-Pooling on fine-grained classification
datasets.

本文提出对比重构（ConRec）算法，它通过联合优化对比和自重构损失获得图像表示，并添加了一个自重构任务和一个在对比学习任务中的注意机制以改进捕捉图像的细粒度视觉特征，证明该算法在捕捉细粒度视觉特征方面优于 SimCLR。

通过结合对比学习、图像重建和注意力加权汇聚，实现细粒度视觉表征

Towards Fine-grained Visual Representations by Combining Contrastive  Learning with Image Reconstruction and Attention-weighted Pooling

Image retrieval with natural language feedback offers the promise of catalog
search based on fine-grained visual features that go beyond objects and binary
attributes, facilitating real-world applications such as e-commerce. Our
Modality-Agnostic Attention Fusion (MAAF) model combines image and text
features and outperforms existing approaches on two visual search with
modifying phrase datasets, Fashion IQ and CSS, and performs competitively on a
dataset with only single-word modifications, Fashion200k. We also introduce two
new challenging benchmarks adapted from Birds-to-Words and Spot-the-Diff, which
provide new settings with rich language inputs, and we show that our approach
without modification outperforms strong baselines. To better understand our
model, we conduct detailed ablations on Fashion IQ and provide visualizations
of the surprising phenomenon of words avoiding "attending" to the image region
they refer to.

本研究采用自然语言反馈的图像检索方法，结合图像与文本特征实现细粒度视觉搜索并提出了 MAAF 模型，在 Fashion IQ 和 CSS 数据集中比现有方法表现更优，同时在 Fashion200k 数据集中也取得竞争性表现。此外，我们还提出了两个适用于丰富语言输入的新挑战基准，并通过实验证明该方法在不修改时优于强基线。最后我们在 Fashion IQ 上进行了深入细致的分析和可视化，揭示了单词避免 “关注” 他们所指图像区域的惊人现象