Visual grounding, which aims to ground a visual region via natural language,
is a task that heavily relies on cross-modal alignment. Existing works utilized
uni-modal pre-trained models to transfer visual/linguistic knowledge separately
while ignoring the multimodal corresponding information. Motivated by recent
advancements in contrastive language-image pre-training and low-rank adaptation
(LoRA) methods, we aim to solve the grounding task based on multimodal
pre-training. However, there exists significant task gaps between pre-training
and grounding. Therefore, to address these gaps, we propose a concise and
efficient hierarchical multimodal fine-grained modulation framework, namely
HiVG. Specifically, HiVG consists of a multi-layer adaptive cross-modal bridge
and a hierarchical multimodal low-rank adaptation (Hi LoRA) paradigm. The
cross-modal bridge can address the inconsistency between visual features and
those required for grounding, and establish a connection between multi-level
visual and text features. Hi LoRA prevents the accumulation of perceptual
errors by adapting the cross-modal features from shallow to deep layers in a
hierarchical manner. Experimental results on five datasets demonstrate the
effectiveness of our approach and showcase the significant grounding
capabilities as well as promising energy efficiency advantages. The project
page: this https URL

通过多模态预训练的方法，我们提出了一个简洁高效的分层多模态细粒度调制框架 HiVG，用于解决通过自然语言对视觉区域进行视觉定位的任务，并且在五个数据集上的实验证明了该方法的有效性和显著的视觉定位能力及能源效率优势。