Web-crawled image-text pairs are inherently noisy. Prior studies demonstrate
that semantically aligning and enriching textual descriptions of these pairs
can significantly enhance model training across various vision-language tasks,
particularly text-to-image generation. However, large-scale investigations in
this area remain predominantly closed-source. Our paper aims to bridge this
community effort, leveraging the powerful and \textit{open-sourced} LLaMA-3, a
GPT-4 level LLM. Our recaptioning pipeline is simple: first, we fine-tune a
LLaMA-3-8B powered LLaVA-1.5 and then employ it to recaption 1.3 billion images
from the DataComp-1B dataset. Our empirical results confirm that this enhanced
dataset, Recap-DataComp-1B, offers substantial benefits in training advanced
vision-language models. For discriminative models like CLIP, we observe
enhanced zero-shot performance in cross-modal retrieval tasks. For generative
models like text-to-image Diffusion Transformers, the generated images exhibit
a significant improvement in alignment with users' text instructions,
especially in following complex queries. Our project page is
this https URL

通过使用 LLaMA-3 构建的 recaptioning pipeline，我们提供了一个改进的数据集 Recap-DataComp-1B，可以显著改善高级视觉语言模型的训练，包括 CLIP 和 text-to-image Diffusion Transformers。

利用 LLaMA-3 对数十亿网络图像进行重新字幕

What If We Recaption Billions of Web Images with LLaMA-3?

Scene graph generation (SGG) is designed to extract (subject, predicate,
object) triplets in images. Recent works have made a steady progress on SGG,
and provide useful tools for high-level vision and language understanding.
However, due to the data distribution problems including long-tail distribution
and semantic ambiguity, the predictions of current SGG models tend to collapse
to several frequent but uninformative predicates (e.g., on, at), which limits
practical application of these models in downstream tasks. To deal with the
problems above, we propose a novel Internal and External Data Transfer
(IETrans) method, which can be applied in a plug-and-play fashion and expanded
to large SGG with 1,807 predicate classes. Our IETrans tries to relieve the
data distribution problem by automatically creating an enhanced dataset that
provides more sufficient and coherent annotations for all predicates. By
training on the enhanced dataset, a Neural Motif model doubles the macro
performance while maintaining competitive micro performance. The code and data
are publicly available at this https URL.

本文提出了一种新颖的内外数据传输（ IETrans ）方法来解决场景图生成的两个数据分布问题，即长尾分布和语义歧义 。通过自动创建增强的数据集，该方法为所有谓词提供了更充分和连贯的注释，从而可应用于大规模带有 1,807 个谓词类的 SGG。最终，通过在增强的数据集上训练，使用神经因子模型可以在保持竞争性宏性能的同时，使得宏性能翻倍。