Understanding the deep semantics of images is essential in the era dominated
by social media. However, current research works primarily on the superficial
description of images, revealing a notable deficiency in the systematic
investigation of the inherent deep semantics. In this work, we introduce
DEEPEVAL, a comprehensive benchmark to assess Large Multimodal Models' (LMMs)
capacities of visual deep semantics. DEEPEVAL includes human-annotated dataset
and three progressive subtasks: fine-grained description selection, in-depth
title matching, and deep semantics understanding. Utilizing DEEPEVAL, we
evaluate 9 open-source LMMs and GPT-4V(ision).Our evaluation demonstrates a
substantial gap between the deep semantic comprehension capabilities of
existing LMMs and humans. For example, GPT-4V is 30% behind humans in
understanding deep semantics, even though it achieves human-comparable
performance in image description. Further analysis indicates that the
integration of description texts during the inference process notably enhances
LMMs' ability to perceive deep semantics. Furthermore, our dataset is divided
into multiple categories, and we conducted a more detailed analysis within
these categories.

通过 DEEPEVAL 评估，本研究发现现有的大型多模态模型对深度语义的理解能力与人类存在明显差距，尽管在图像描述方面达到了与人类相媲美的性能。进一步分析表明，推理过程中结合描述文本可以显著增强大型多模态模型对深度语义的感知能力。

大型多模型模型能否揭示图像背后的深层语义？

Can Large Multimodal Models Uncover Deep Semantics Behind Images?

Neural networks are prone to learn easy solutions from superficial statistics
in the data, namely shortcut learning, which impairs generalization and
robustness of models. We propose a data augmentation strategy, named DFM-X,
that leverages knowledge about frequency shortcuts, encoded in Dominant
Frequencies Maps computed for image classification models. We randomly select
X% training images of certain classes for augmentation, and process them by
retaining the frequencies included in the DFMs of other classes. This strategy
compels the models to leverage a broader range of frequencies for
classification, rather than relying on specific frequency sets. Thus, the
models learn more deep and task-related semantics compared to their counterpart
trained with standard setups. Unlike other commonly used augmentation
techniques which focus on increasing the visual variations of training data,
our method targets exploiting the original data efficiently, by distilling
prior knowledge about destructive learning behavior of models from data. Our
experimental results demonstrate that DFM-X improves robustness against common
corruptions and adversarial attacks. It can be seamlessly integrated with other
augmentation techniques to further enhance the robustness of models.

在这篇论文中，我们提出了一种名为 DFM-X 的数据增强策略，它利用了图像分类模型中编码的频率快捷方式的知识，并引导模型利用更广泛的频率范围进行分类，以提高模型的鲁棒性和任务相关的语义，同时我们的实验证明 DFM-X 可与其他增强技术无缝集成，进一步提高模型的鲁棒性。

DFM-X：借助先验知识加强的捷径学习增强

DFM-X: Augmentation by Leveraging Prior Knowledge of Shortcut Learning

We demonstrate the need and potential of systematically integrated vision and
semantics solutions for visual sensemaking in the backdrop of autonomous
driving. A general neurosymbolic method for online visual sensemaking using
answer set programming (ASP) is systematically formalised and fully
implemented. The method integrates state of the art in visual computing, and is
developed as a modular framework that is generally usable within hybrid
architectures for realtime perception and control. We evaluate and demonstrate
with community established benchmarks KITTIMOD, MOT-2017, and MOT-2020. As
use-case, we focus on the significance of human-centred visual sensemaking --
e.g., involving semantic representation and explainability, question-answering,
commonsense interpolation -- in safety-critical autonomous driving situations.
The developed neurosymbolic framework is domain-independent, with the case of
autonomous driving designed to serve as an exemplar for online visual
sensemaking in diverse cognitive interaction settings in the backdrop of select
human-centred AI technology design considerations.
Keywords: Cognitive Vision, Deep Semantics, Declarative Spatial Reasoning,
Knowledge Representation and Reasoning, Commonsense Reasoning, Visual
Abduction, Answer Set Programming, Autonomous Driving, Human-Centred Computing
and Design, Standardisation in Driving Technology, Spatial Cognition and AI.

本文系统地阐述了自动驾驶背景下，系统集成视觉和语义解决方案用于视觉意义建模的需求和潜力，提出了一种通用的神经符号方法，使用 ASP 进行在线视觉意义建模，该方法整合了最先进的视觉计算技术和深度语义学习，是一个可在混合体系结构下用于实时感知和控制的模块化框架，在 KITTIMOD，MOT-2017 和 MOT-2020 等声誉良好的基准测试中进行了评估和论证。