Recent Vision-Language Pre-training (VLP) models have demonstrated
significant advancements. Nevertheless, these models heavily rely on image-text
pairs that capture only coarse and global information of an image, leading to a
limitation in their regional understanding ability. In this work, we introduce
\textbf{RegionVLM}, equipped with explicit regional modeling capabilities,
allowing them to understand user-indicated image regions. To achieve this, we
design a simple yet innovative architecture, requiring no modifications to the
model architecture or objective function. Additionally, we leverage a dataset
that contains a novel source of information, namely Localized Narratives, which
has been overlooked in previous VLP research. Our experiments demonstrate that
our single generalist model not only achieves an interactive dialogue system
but also exhibits superior performance on various zero-shot region
understanding tasks, without compromising its ability for global image
understanding.

通过引入具有明确区域建模能力的 RegionVLM 模型，并利用包含区域信息的 Localized Narratives 数据集，我们的实验表明，我们的单一通用模型不仅实现了交互式对话系统，还在各种零样本区域理解任务上展现出了卓越的性能，而不会损害其对全局图像的理解能力。

在视觉 - 语言模型中实现交互式区域理解

Toward Interactive Regional Understanding in Vision-Large Language  Models

Localized Narratives is a dataset with detailed natural language descriptions
of images paired with mouse traces that provide a sparse, fine-grained visual
grounding for phrases. We propose TReCS, a sequential model that exploits this
grounding to generate images. TReCS uses descriptions to retrieve segmentation
masks and predict object labels aligned with mouse traces. These alignments are
used to select and position masks to generate a fully covered segmentation
canvas; the final image is produced by a segmentation-to-image generator using
this canvas. This multi-step, retrieval-based approach outperforms existing
direct text-to-image generation models on both automatic metrics and human
evaluations: overall, its generated images are more photo-realistic and better
match descriptions.

提出了基于 Local Narratives 数据集和 TReCS 模型的图像文本生成方法，在视觉的基础上，利用分割掩模等技术实现了更好的图像合成效果。

基于细粒度用户注意力的文本到图像生成

Text-to-Image Generation Grounded by Fine-Grained User Attention

We propose Localized Narratives, a new form of multimodal image annotations
connecting vision and language. We ask annotators to describe an image with
their voice while simultaneously hovering their mouse over the region they are
describing. Since the voice and the mouse pointer are synchronized, we can
localize every single word in the description. This dense visual grounding
takes the form of a mouse trace segment per word and is unique to our data. We
annotated 849k images with Localized Narratives: the whole COCO, Flickr30k, and
ADE20K datasets, and 671k images of Open Images, all of which we make publicly
available. We provide an extensive analysis of these annotations showing they
are diverse, accurate, and efficient to produce. We also demonstrate their
utility on the application of controlled image captioning.

我们提出了一种新的多模态图像标注方法，称为定位叙述，将视觉和语言进行连接。通过请求标注者在将鼠标指针悬停在要描述的区域的同时用语音来描述图像，我们实现了对每个单词进行定位。该方法经过全面的分析和外部数据验证，具有高度准确性和生产效率，并且在受控图像字幕应用程序中具有实用性。