Contrastive Language-Image Pre-training (CLIP) excels in multimodal tasks such as image-text retrieval and zero-shot classification but struggles with fine-grained understanding due to its focus on coarse-grained short captions. To address this, we propose Fine-Grained CLIP (FG-CLIP), which enhances fine-grained understanding through three key innovations. First, we leverage large multimodal models to generate 1.6 billion long caption-image pairs for capturing global-level semantic details. Second, a high-quality dataset is constructed with 12 million images and 40 million region-specific bounding boxes aligned with detailed captions to ensure precise, context-rich representations. Third, 10 million hard fine-grained negative samples are incorporated to improve the model's ability to distinguish subtle semantic differences. We construct a comprehensive dataset, termed FgGRN, by integrating high-quality region-specific annotations with challenging fine-grained negative samples. Corresponding training methods are meticulously designed for these data. Extensive experiments demonstrate that FG-CLIP outperforms the original CLIP and other state-of-the-art methods across various downstream tasks, including fine-grained understanding, open-vocabulary object detection, image-text retrieval, and general multimodal benchmarks. These results highlight FG-CLIP's effectiveness in capturing fine-grained image details and improving overall model performance. The related data, code, and models are available at https://github.com/360CVGroup/FG-CLIP.

本研究解决了CLIP在细粒度理解方面的不足，通过生成16亿对长文本和图像，以捕捉全局语义信息，并构建包含1200万张图像和4000万个细分包围框的高质量数据集。此外，还加入1000万个困难的细粒度负样本，提升模型区分细微语义差异的能力。实验结果表明，FG-CLIP在多个下游任务中超越了原 CLIP和其他先进方法，展示了其在捕捉细粒度图像细节和提升整体模型性能方面的有效性。

FG-CLIP：细粒度视觉与文本对齐