Large Vision-Language models (VLMs) have demonstrated strong reasoning
capabilities in tasks requiring a fine-grained understanding of literal images
and text, such as visual question-answering or visual entailment. However,
there has been little exploration of these models' capabilities when presented
with images and captions containing figurative phenomena such as metaphors or
humor, the meaning of which is often implicit. To close this gap, we propose a
new task and a high-quality dataset: Visual Figurative Language Understanding
with Textual Explanations (V-FLUTE). We frame the visual figurative language
understanding problem as an explainable visual entailment task, where the model
has to predict whether the image (premise) entails a claim (hypothesis) and
justify the predicted label with a textual explanation. Using a human-AI
collaboration framework, we build a high-quality dataset, V-FLUTE, that
contains 6,027 <image, claim, label, explanation> instances spanning five
diverse multimodal figurative phenomena: metaphors, similes, idioms, sarcasm,
and humor. The figurative phenomena can be present either in the image, the
caption, or both. We further conduct both automatic and human evaluations to
assess current VLMs' capabilities in understanding figurative phenomena.

使用视觉和语言模型进行图像和文字的理解，探索其在理解比喻和幽默等比喻现象方面的能力，并提出了一个新的任务和数据集，Visual Figurative Language Understanding with Textual Explanations（V-FLUTE）。