Generating detailed and accurate descriptions for specific regions in images and videos remains a fundamental challenge for vision-language models. We introduce the Describe Anything Model (DAM), a model designed for detailed localized captioning (DLC). DAM preserves both local details and global context through two key innovations: a focal prompt, which ensures high-resolution encoding of targeted regions, and a localized vision backbone, which integrates precise localization with its broader context. To tackle the scarcity of high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark designed to evaluate DLC without relying on reference captions. DAM sets new state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and detailed multi-sentence localized image and video captioning.

本研究针对视觉-语言模型在图像和视频特定区域生成详细准确描述的挑战，提出了描述任何内容模型（DAM），通过聚焦提示和局部视觉骨干实现局部细节与全局上下文的结合。该模型在七个基准测试中刷新了局部字幕生成的最新记录，显示了其在关键词、短语和多句详尽字幕生成方面的显著进展。

描述任何内容：详细的局部图像和视频字幕生成