Recent advancements in large multimodal language models have demonstrated
remarkable proficiency across a wide range of tasks. Yet, these models still
struggle with understanding the nuances of human humor through juxtaposition,
particularly when it involves nonlinear narratives that underpin many jokes and
humor cues. This paper investigates this challenge by focusing on comics with
contradictory narratives, where each comic consists of two panels that create a
humorous contradiction. We introduce the YesBut benchmark, which comprises
tasks of varying difficulty aimed at assessing AI's capabilities in recognizing
and interpreting these comics, ranging from literal content comprehension to
deep narrative reasoning. Through extensive experimentation and analysis of
recent commercial or open-sourced large (vision) language models, we assess
their capability to comprehend the complex interplay of the narrative humor
inherent in these comics. Our results show that even state-of-the-art models
still lag behind human performance on this task. Our findings offer insights
into the current limitations and potential improvements for AI in understanding
human creative expressions.

通过对具有矛盾叙事的漫画展开研究，本文引入了 YesBut 评估基准，旨在评估人工智能在识别和解释具有幽默矛盾的漫画方面的能力。通过广泛的实验和分析，我们发现即使是最先进的模型在此任务上仍然落后于人类表现，为人工智能在理解人类创造性表达方面的当前限制和潜在改进提供了见解。

解读并列: AI 模型能理解幽默的矛盾吗

Cracking the Code of Juxtaposition: Can AI Models Understand the  Humorous Contradictions

Visual narrative is often a combination of explicit information and judicious
omissions, relying on the viewer to supply missing details. In comics, most
movements in time and space are hidden in the "gutters" between panels. To
follow the story, readers logically connect panels together by inferring unseen
actions through a process called "closure". While computers can now describe
what is explicitly depicted in natural images, in this paper we examine whether
they can understand the closure-driven narratives conveyed by stylized artwork
and dialogue in comic book panels. We construct a dataset, COMICS, that
consists of over 1.2 million panels (120 GB) paired with automatic textbox
transcriptions. An in-depth analysis of COMICS demonstrates that neither text
nor image alone can tell a comic book story, so a computer must understand both
modalities to keep up with the plot. We introduce three cloze-style tasks that
ask models to predict narrative and character-centric aspects of a panel given
n preceding panels as context. Various deep neural architectures underperform
human baselines on these tasks, suggesting that COMICS contains fundamental
challenges for both vision and language.

本研究构建了一个新的数据集 COMICS，通过对其深入分析得出结论：计算机只有同时理解文本与图像，才能理解闭合驱动的漫画故事，本文提出了三个任务，并在此基础上比较了不同的深度神经网络结构，研究表明该数据集的视觉以及语言的基础难点都需要在未来持续深研究。