Large-scale Vision-Language Models (VLMs) have demonstrated exceptional
performance in natural vision tasks, motivating researchers across domains to
explore domain-specific VLMs. However, the construction of powerful
domain-specific VLMs demands vast amounts of annotated data, substantial
electrical energy, and computing resources, primarily accessible to industry,
yet hindering VLM research in academia. To address this challenge and foster
sustainable and equitable VLM research, we present the Generalized Domain
Prompt Learning (GDPL) framework. GDPL facilitates the transfer of VLMs' robust
recognition capabilities from natural vision to specialized domains, without
the need for extensive data or resources. By leveraging small-scale
domain-specific foundation models and minimal prompt samples, GDPL empowers the
language branch with domain knowledge through quaternion networks, uncovering
cross-modal relationships between domain-specific vision features and natural
vision-based contextual embeddings. Simultaneously, GDPL guides the vision
branch into specific domains through hierarchical propagation of generated
vision prompt features, grounded in well-matched vision-language relations.
Furthermore, to fully harness the domain adaptation potential of VLMs, we
introduce a novel low-rank adaptation approach. Extensive experiments across
diverse domains like remote sensing, medical imaging, geology, Synthetic
Aperture Radar, and fluid dynamics, validate the efficacy of GDPL,
demonstrating its ability to achieve state-of-the-art domain recognition
performance in a prompt learning paradigm. Our framework paves the way for
sustainable and inclusive VLM research, transcending the barriers between
academia and industry.

大规模视觉语言模型（VLMs）在自然视觉任务中表现出色，推动跨领域的研究者探索特定领域的 VLMs。然而，构建强大的特定领域 VLMs 需要大量的注释数据、大量的电能和计算资源，这些主要是工业界可用的，却制约了学术界对 VLMs 的研究。为了解决这个挑战，促进可持续和公平的 VLMs 研究，我们提出了广义领域提示学习（GDPL）框架。GDPL 通过小规模的特定领域基础模型和最少的提示样本，使 VLMs 的强大识别能力从自然视觉转移到特定领域，而无需大量数据或资源。通过四元网络，GDPL 利用特定领域的基础模型和最少的提示样本为语言分支赋予领域知识，揭示特定领域视觉特征与自然视觉上下文嵌入之间的跨模态关系。同时，GDPL 通过生成的视觉提示特征的分层传导，将视觉分支引向特定领域，建立与视觉语言关系匹配的基础。此外，为了充分发挥 VLMs 的领域自适应潜力，我们引入了一种新颖的低秩自适应方法。对遥感、医学成像、地质学、合成孔径雷达和流体动力学等不同领域的大量实验证明了 GDPL 的有效性，并展示了在提示学习范式下实现最先进的领域识别性能的能力。我们的框架为可持续和包容性的 VLMs 研究铺平了道路，消除了学术界与工业界之间的障碍。