Panoptic narrative grounding (PNG), whose core target is fine-grained image-text alignment, requires a panoptic segmentation of referred objects given a narrative caption. Previous discriminative methods achieve only weak or coarse-grained alignment by panoptic segmentation pretraining or CLIP model adaptation. Given the recent progress of text-to-image Diffusion models, several works have shown their capability to achieve fine-grained image-text alignment through cross-attention maps and improved general segmentation performance. However, the direct use of phrase features as static prompts to apply frozen Diffusion models to the PNG task still suffers from a large task gap and insufficient vision-language interaction, yielding inferior performance. Therefore, we propose an Extractive-Injective Phrase Adapter (EIPA) bypass within the Diffusion UNet to dynamically update phrase prompts with image features and inject the multimodal cues back, which leverages the fine-grained image-text alignment capability of Diffusion models more sufficiently. In addition, we also design a Multi-Level Mutual Aggregation (MLMA) module to reciprocally fuse multi-level image and phrase features for segmentation refinement. Extensive experiments on the PNG benchmark show that our method achieves new state-of-the-art performance.

本研究解决了全景叙事对接中图像与文本之间的细粒度对齐问题。通过提出一种提取-注入短语适配器(EIPA)，该方法动态更新短语提示以更好地与图像特征交互，显著提高了扩散模型在此任务上的表现。此外，设计的多层互聚合(MLMA)模块进一步增强了多层次特征的融合和细分。实验表明，该方法在基准测试中达到了新的最先进性能。

动态提示冻结文本至图像扩散模型以实现全景叙事对接