Prompt learning has become a prevalent strategy for adapting vision-language
foundation models to downstream tasks. As large language models (LLMs) have
emerged, recent studies have explored the use of category-related descriptions
as input to enhance prompt effectiveness. Nevertheless, conventional
descriptions fall short of structured information that effectively represents
the interconnections among entities or attributes linked to a particular
category. To address this limitation and prioritize harnessing structured
knowledge, this paper advocates for leveraging LLMs to build a graph for each
description to model the entities and attributes describing the category, as
well as their correlations. Preexisting prompt tuning methods exhibit
inadequacies in managing this structured knowledge. Consequently, we propose a
novel approach called Hierarchical Prompt Tuning (HPT), which enables
simultaneous modeling of both structured and conventional linguistic knowledge.
Specifically, we introduce a relationship-guided attention module to capture
pair-wise associations among entities and attributes for low-level prompt
learning. In addition, by incorporating high-level and global-level prompts
modeling overall semantics, the proposed hierarchical structure forges
cross-level interlinks and empowers the model to handle more complex and
long-term relationships. Extensive experiments demonstrate that our HPT shows
strong effectiveness and generalizes much better than existing SOTA methods.
Our code is available at this https URL

本研究提出一种称为分层提示调整的新方法，利用大语言模型（LLMs）建立描述每个类别的图形模型，以模拟描述类别的实体和属性及其相关性，并通过引入关系引导注意模块来捕捉实体和属性之间的关联，以及通过结合整体语义建模来处理更复杂和长期的关系。实验结果表明，我们的方法具有很强的效果，并且比现有的最先进方法具有更好的泛化性能。