Vision-language datasets are vital for both text-to-image (T2I) and
image-to-text (I2T) research. However, current datasets lack descriptions with
fine-grained detail that would allow for richer associations to be learned by
models. To fill the gap, we introduce Descriptions of Connected and Contrasting
Images (DOCCI), a dataset with long, human-annotated English descriptions for
15k images that were taken, curated and donated by a single researcher intent
on capturing key challenges such as spatial relations, counting, text
rendering, world knowledge, and more. We instruct human annotators to create
comprehensive descriptions for each image; these average 136 words in length
and are crafted to clearly distinguish each image from those that are related
or similar. Each description is highly compositional and typically encompasses
multiple challenges. Through both quantitative and qualitative analyses, we
demonstrate that DOCCI serves as an effective training resource for
image-to-text generation -- a PaLI 5B model finetuned on DOCCI shows equal or
superior results compared to highly-performant larger models like LLaVA-1.5 7B
and InstructBLIP 7B. Furthermore, we show that DOCCI is a useful testbed for
text-to-image generation, highlighting the limitations of current text-to-image
models in capturing long descriptions and fine details.

通过介绍 Descriptions of Connected and Contrasting Images (DOCCI) 数据集，本文表明 DOCCI 是一个有效的图像到文本生成的训练资源，并展示了当前文本到图像模型在捕捉长描述和细节方面的局限性。