Traditional Bag-of-visual Words (BoWs) model is commonly generated with many
steps including local feature extraction, codebook generation, and feature
quantization, etc. Those steps are relatively independent with each other and
are hard to be jointly optimized. Moreover, the dependency on hand-crafted
local feature makes BoWs model not effective in conveying high-level semantics.
These issues largely hinder the performance of BoWs model in large-scale image
applications. To conquer these issues, we propose an End-to-End BoWs
(E$^2$BoWs) model based on Deep Convolutional Neural Network (DCNN). Our model
takes an image as input, then identifies and separates the semantic objects in
it, and finally outputs the visual words with high semantic discriminative
power. Specifically, our model firstly generates Semantic Feature Maps (SFMs)
corresponding to different object categories through convolutional layers, then
introduces Bag-of-Words Layers (BoWL) to generate visual words for each
individual feature map. We also introduce a novel learning algorithm to
reinforce the sparsity of the generated E$^2$BoWs model, which further ensures
the time and memory efficiency. We evaluate the proposed E$^2$BoWs model on
several image search datasets including CIFAR-10, CIFAR-100, MIRFLICKR-25K and
NUS-WIDE. Experimental results show that our method achieves promising accuracy
and efficiency compared with recent deep learning based retrieval works.

本文提出了一种基于深度卷积神经网络的端到端 BoWs 模型 (E$^2$BoWs)，并介绍了一种新的学习算法来增强生成的 E$^2$BoWs 模型的稀疏性，以进一步确保时间和内存效率。在多个图像搜索数据集上的实验结果表明，与最近的基于深度学习的检索方法相比，我们的模型在准确性和效率上都取得了很好的表现。