Vision and vision-language applications of neural networks, such as image
classification and captioning, rely on large-scale annotated datasets that
require non-trivial data-collecting processes. This time-consuming endeavor
hinders the emergence of large-scale datasets, limiting researchers and
practitioners to a small number of choices. Therefore, we seek more efficient
ways to collect and annotate images. Previous initiatives have gathered
captions from HTML alt-texts and crawled social media postings, but these data
sources suffer from noise, sparsity, or subjectivity. For this reason, we turn
to commercial shopping websites whose data meet three criteria: cleanliness,
informativeness, and fluency. We introduce the Let's Go Shopping (LGS) dataset,
a large-scale public dataset with 15 million image-caption pairs from publicly
available e-commerce websites. When compared with existing general-domain
datasets, the LGS images focus on the foreground object and have less complex
backgrounds. Our experiments on LGS show that the classifiers trained on
existing benchmark datasets do not readily generalize to e-commerce data, while
specific self-supervised visual feature extractors can better generalize.
Furthermore, LGS's high-quality e-commerce-focused images and bimodal nature
make it advantageous for vision-language bi-modal tasks: LGS enables
image-captioning models to generate richer captions and helps text-to-image
generation models achieve e-commerce style transfer.

通过商业购物网站提供的大规模公共数据集‘Let's Go Shopping (LGS)’，我们发现现存基准数据集的分类器在电子商务数据上无法很好地进行泛化，而特定的自我监督视觉特征提取器则可以更好地进行泛化，此外，LGS 数据集的高质量电子商务特定图像和双模态性质，使其对于视觉 - 语言双模态任务具有优势，可以帮助图像生成丰富的标题，并实现电子商务风格的文本到图像转换。

大规模网页图像 - 文本数据集用于视觉概念理解

Let's Go Shopping (LGS) -- Web-Scale Image-Text Dataset for Visual  Concept Understanding

We present Catalog Phrase Grounding (CPG), a model that can associate product
textual data (title, brands) into corresponding regions of product images
(isolated product region, brand logo region) for e-commerce vision-language
applications. We use a state-of-the-art modulated multimodal transformer
encoder-decoder architecture unifying object detection and phrase-grounding. We
train the model in self-supervised fashion with 2.3 million image-text pairs
synthesized from an e-commerce site. The self-supervision data is annotated
with high-confidence pseudo-labels generated with a combination of teacher
models: a pre-trained general domain phrase grounding model (e.g. MDETR) and a
specialized logo detection model. This allows CPG, as a student model, to
benefit from transfer knowledge from these base models combining general-domain
knowledge and specialized knowledge. Beyond immediate catalog phrase grounding
tasks, we can benefit from CPG representations by incorporating them as ML
features into downstream catalog applications that require deep semantic
understanding of products. Our experiments on product-brand matching, a
challenging e-commerce application, show that incorporating CPG representations
into the existing production ensemble system leads to on average 5% recall
improvement across all countries globally (with the largest lift of 11% in a
single country) at fixed 95% precision, outperforming other alternatives
including a logo detection teacher model and ResNet50.

我们提出了 Catalog Phrase Grounding (CPG) 模型，能够将产品的文本数据（标题、品牌）关联到相应的产品图像区域（孤立的产品区域、品牌标志区域），用于电子商务视觉语言应用。我们使用最先进的调制多模态 Transformer 编码器 - 解码器架构，统一了物体检测和短语关联。通过从电子商务网站合成的 230 万个图像文本对进行自监督训练。自监督数据使用由一些教师模型（例如，预训练的通用领域短语关联模型和专门的标志检测模型）生成的高置信度伪标签进行注释。这使得 CPG 作为学生模型可以从这些基础模型的转移知识中受益，结合了通用领域知识和专业知识。除了立即的目录短语关联任务之外，我们将 CPG 表示引入到需要深度语义理解产品的下游目录应用中作为机器学习特征，从中受益。我们在一个具有挑战性的电子商务应用（产品 - 品牌匹配）上的实验表明，将 CPG 表示引入到现有的生产集成系统中，在固定的 95% 精度下，全球性能平均提升 5％的召回率（在一个单一的国家中提升 11％），超越了其他可选方案，包括标志检测教师模型和 ResNet50。

目录短语引导（CPG）：用于电子商务视觉语言应用中将产品文本属性与产品图像关联

Catalog Phrase Grounding (CPG): Grounding of Product Textual Attributes  in Product Images for e-commerce Vision-Language Applications

Large-scale pre-trained text-image models with dual-encoder architectures
(such as CLIP) are typically adopted for various vision-language applications,
including text-image retrieval. However,these models are still less practical
on edge devices or for real-time situations, due to the substantial indexing
and inference time and the large consumption of computational resources.
Although knowledge distillation techniques have been widely utilized for
uni-modal model compression, how to expand them to the situation when the
numbers of modalities and teachers/students are doubled has been rarely
studied. In this paper, we conduct comprehensive experiments on this topic and
propose the fully-Connected knowledge interaction graph (Cona) technique for
cross-modal pre-training distillation. Based on our findings, the resulting
ConaCLIP achieves SOTA performances on the widely-used Flickr30K and MSCOCO
benchmarks under the lightweight setting. An industry application of our method
on an e-commercial platform further demonstrates the significant effectiveness
of ConaCLIP.

本文探讨了如何将知识蒸馏技术扩展到双模态模型情况，并提出了一个全连接知识交互图（Cona）技术，用于跨模态预训练蒸馏，实验证明该方法具有显著的效果。