Creativity is an indispensable part of human cognition and also an inherent
part of how we make sense of the world. Metaphorical abstraction is fundamental
in communicating creative ideas through nuanced relationships between abstract
concepts such as feelings. While computer vision benchmarks and approaches
predominantly focus on understanding and generating literal interpretations of
images, metaphorical comprehension of images remains relatively unexplored.
Towards this goal, we introduce MetaCLUE, a set of vision tasks on visual
metaphor. We also collect high-quality and rich metaphor annotations (abstract
objects, concepts, relationships along with their corresponding object boxes)
as there do not exist any datasets that facilitate the evaluation of these
tasks. We perform a comprehensive analysis of state-of-the-art models in vision
and language based on our annotations, highlighting strengths and weaknesses of
current approaches in visual metaphor Classification, Localization,
Understanding (retrieval, question answering, captioning) and gEneration
(text-to-image synthesis) tasks. We hope this work provides a concrete step
towards developing AI systems with human-like creative capabilities.

本文介绍了 MetaCLUE—— 一组用于视觉隐喻的视觉任务，以及收集的高质量且丰富的比喻注释数据集，并对基于注释的最先进视觉和语言模型进行了全面的分析，重点介绍了当前方法在视觉隐喻分类、定位、理解和生成方面的优势和弱点，旨在为开发具有类人创造力的 AI 系统迈出具体的一步。