Hierarchical semantic structures naturally exist in an image dataset, in
which several semantically relevant image clusters can be further integrated
into a larger cluster with coarser-grained semantics. Capturing such structures
with image representations can greatly benefit the semantic understanding on
various downstream tasks. Existing contrastive representation learning methods
lack such an important model capability. In addition, the negative pairs used
in these methods are not guaranteed to be semantically distinct, which could
further hamper the structural correctness of learned image representations. To
tackle these limitations, we propose a novel contrastive learning framework
called Hierarchical Contrastive Selective Coding (HCSC). In this framework, a
set of hierarchical prototypes are constructed and also dynamically updated to
represent the hierarchical semantic structures underlying the data in the
latent space. To make image representations better fit such semantic
structures, we employ and further improve conventional instance-wise and
prototypical contrastive learning via an elaborate pair selection scheme. This
scheme seeks to select more diverse positive pairs with similar semantics and
more precise negative pairs with truly distinct semantics. On extensive
downstream tasks, we verify the superior performance of HCSC over
state-of-the-art contrastive methods, and the effectiveness of major model
components is proved by plentiful analytical studies. We build a comprehensive
model zoo in Sec. D. Our source code and model weights are available at
this https URL

提出了一种新的对比学习框架 HCSC，可以通过构建并动态更新一组层次结构原型来表征图片数据中的层次语义结构，并且通过一种精心设计的对对选择方案来让图像表示更好地适应这些语义结构。在广泛的下游任务中，证明了 HCSC 优于现有对比方法，其重要模型组件的有效性通过大量的分析研究得到证明。