Understanding the deep semantics of images is essential in the era dominated
by social media. However, current research works primarily on the superficial
description of images, revealing a notable deficiency in the systematic
investigation of the inherent deep semantics. In this work, we introduce
DEEPEVAL, a comprehensive benchmark to assess Large Multimodal Models' (LMMs)
capacities of visual deep semantics. DEEPEVAL includes human-annotated dataset
and three progressive subtasks: fine-grained description selection, in-depth
title matching, and deep semantics understanding. Utilizing DEEPEVAL, we
evaluate 9 open-source LMMs and GPT-4V(ision).Our evaluation demonstrates a
substantial gap between the deep semantic comprehension capabilities of
existing LMMs and humans. For example, GPT-4V is 30% behind humans in
understanding deep semantics, even though it achieves human-comparable
performance in image description. Further analysis indicates that the
integration of description texts during the inference process notably enhances
LMMs' ability to perceive deep semantics. Furthermore, our dataset is divided
into multiple categories, and we conducted a more detailed analysis within
these categories.

通过 DEEPEVAL 评估，本研究发现现有的大型多模态模型对深度语义的理解能力与人类存在明显差距，尽管在图像描述方面达到了与人类相媲美的性能。进一步分析表明，推理过程中结合描述文本可以显著增强大型多模态模型对深度语义的感知能力。