Object categories are typically organized into a multi-granularity taxonomic
hierarchy. When classifying categories at different hierarchy levels,
traditional uni-modal approaches focus primarily on image features, revealing
limitations in complex scenarios. Recent studies integrating Vision-Language
Models (VLMs) with class hierarchies have shown promise, yet they fall short of
fully exploiting the hierarchical relationships. These efforts are constrained
by their inability to perform effectively across varied granularity of
categories. To tackle this issue, we propose a novel framework (HGCLIP) that
effectively combines CLIP with a deeper exploitation of the Hierarchical class
structure via Graph representation learning. We explore constructing the class
hierarchy into a graph, with its nodes representing the textual or image
features of each category. After passing through a graph encoder, the textual
features incorporate hierarchical structure information, while the image
features emphasize class-aware features derived from prototypes through the
attention mechanism. Our approach demonstrates significant improvements on both
generic and fine-grained visual recognition benchmarks. Our codes are fully
available at this https URL

我们提出了一种将 CLIP 与图表示学习相结合以更深入地利用分层类结构的新框架（HGCLIP），通过将类层次结构构建成一个图，其中其节点代表每个类别的文本或图像特征，通过图编码器，文本特征结合了分层结构信息，而图像特征通过注意机制强调了从原型派生的类别感知特征，我们的方法在通用和细粒度视觉识别基准上均取得了显著的改进。