Online content is filled with logos, from ads and social media posts to
website branding and product placements. Consequently, these logos are
prevalent in the extensive web-scraped datasets used to pretrain
Vision-Language Models, which are used for a wide array of tasks (content
moderation, object classification). While these models have been shown to learn
harmful correlations in various tasks, whether these correlations include logos
remains understudied. Understanding this is especially important due to logos
often being used by public-facing entities like brands and government agencies.
To that end, we develop SLANT: A Spurious Logo ANalysis Toolkit. Our key
finding is that some logos indeed lead to spurious incorrect predictions, for
example, adding the Adidas logo to a photo of a person causes a model classify
the person as greedy. SLANT contains a semi-automatic mechanism for mining such
"spurious" logos. The mechanism consists of a comprehensive logo bank,
CC12M-LogoBank, and an algorithm that searches the bank for logos that VLMs
spuriously correlate with a user-provided downstream recognition target. We
uncover various seemingly harmless logos that VL models correlate 1) with
negative human adjectives 2) with the concept of `harmlessness'; causing models
to misclassify harmful online content as harmless, and 3) with user-provided
object concepts; causing lower recognition accuracy on ImageNet zero-shot
classification. Furthermore, SLANT's logos can be seen as effective attacks
against foundational models; an attacker could place a spurious logo on harmful
content, causing the model to misclassify it as harmless. This threat is
alarming considering the simplicity of logo attacks, increasing the attack
surface of VL models. As a defense, we include in our Toolkit two effective
mitigation strategies that seamlessly integrate with zero-shot inference of
foundation models.

在线内容中存在许多标志，这些标志广泛应用于用于预训练视觉语言模型的大量网络抓取的数据集中。通过开发了一个工具集 SLANT，我们发现一些标志确实会导致虚假的错误预测。SLANT 的标志可被视为对基础模型的有效攻击，为了应对这一威胁，我们还提供了两种有效的缓解策略。

SLANT: 虚假标志分析工具

SLANT: Spurious Logo ANalysis Toolkit

As an important task in multimodal context understanding, Text-VQA (Visual
Question Answering) aims at question answering through reading text information
in images. It differentiates from the original VQA task as Text-VQA requires
large amounts of scene-text relationship understanding, in addition to the
cross-modal grounding capability. In this paper, we propose Localize, Group,
and Select (LOGOS), a novel model which attempts to tackle this problem from
multiple aspects. LOGOS leverages two grounding tasks to better localize the
key information of the image, utilizes scene text clustering to group
individual OCR tokens, and learns to select the best answer from different
sources of OCR (Optical Character Recognition) texts. Experiments show that
LOGOS outperforms previous state-of-the-art methods on two Text-VQA benchmarks
without using additional OCR annotation data. Ablation studies and analysis
demonstrate the capability of LOGOS to bridge different modalities and better
understand scene text.

本文提出了一种名为 Localize, Group, and Select (LOGOS) 的模型，它利用场景文本聚类和光学字符识别（OCR）技术来更好地定位图像的关键信息、实现跨模态理解，并从不同来源的 OCR 文本中选择最佳答案，成功解决了多模态上下文理解中的 Text-VQA 问题，实验表明，该模型在两个 Text-VQA 基准测试上表现优于其他方法。