Learning discriminative image feature embeddings is of great importance to
visual recognition. To achieve better feature embeddings, most current methods
focus on designing different network structures or loss functions, and the
estimated feature embeddings are usually only related to the input images. In
this paper, we propose Memory-based Neighbourhood Embedding (MNE) to enhance a
general CNN feature by considering its neighbourhood. The method aims to solve
two critical problems, i.e., how to acquire more relevant neighbours in the
network training and how to aggregate the neighbourhood information for a more
discriminative embedding. We first augment an episodic memory module into the
network, which can provide more relevant neighbours for both training and
testing. Then the neighbours are organized in a tree graph with the target
instance as the root node. The neighbourhood information is gradually
aggregated to the root node in a bottom-up manner, and aggregation weights are
supervised by the class relationships between the nodes. We apply MNE on image
search and few shot learning tasks. Extensive ablation studies demonstrate the
effectiveness of each component, and our method significantly outperforms the
state-of-the-art approaches.

本文提出了一种记忆邻域嵌入方法（Memory-based Neighbourhood Embedding，MNE），其将邻域信息考虑进卷积神经网络（CNN）特征嵌入，通过增加体现输入图像相关性的记忆模块并使用一个树状图形式的邻域结构来聚合邻域信息以获得更为判别性的嵌入；该方法在图像搜索和少样本学习任务上得到应用，表现出较高性能。