Sarcasm generation has been investigated in previous studies by considering
it as a text-to-text generation problem, i.e., generating a sarcastic sentence
for an input sentence. In this paper, we study a new problem of cross-modal
sarcasm generation (CMSG), i.e., generating a sarcastic description for a given
image. CMSG is challenging as models need to satisfy the characteristics of
sarcasm, as well as the correlation between different modalities. In addition,
there should be some inconsistency between the two modalities, which requires
imagination. Moreover, high-quality training data is insufficient. To address
these problems, we take a step toward generating sarcastic descriptions from
images without paired training data and propose an
Extraction-Generation-Ranking based Modular method (EGRM) for cross-model
sarcasm generation. Specifically, EGRM first extracts diverse information from
an image at different levels and uses the obtained image tags, sentimental
descriptive caption, and commonsense-based consequence to generate candidate
sarcastic texts. Then, a comprehensive ranking algorithm, which considers
image-text relation, sarcasticness, and grammaticality, is proposed to select a
final text from the candidate texts. Human evaluation at five criteria on a
total of 1200 generated image-text pairs from eight systems and auxiliary
automatic evaluation show the superiority of our method.

本研究通过运用 EGRM 模块化方法和排序算法，提出了一种在缺少配对数据的情况下从图像生成讽刺描述的新方法，并通过人工评估和辅助自动评估证明了其优越性。

如何更有趣地描述图片？走向跨模态反讽生成的模块化方法

How to Describe Images in a More Funny Way? Towards a Modular Approach to Cross-Modal Sarcasm Generation

Recent methods for embodied instruction following are typically trained
end-to-end using imitation learning. This often requires the use of expert
trajectories and low-level language instructions. Such approaches assume that
neural states will integrate multimodal semantics to perform state tracking,
building spatial memory, exploration, and long-term planning. In contrast, we
propose a modular method with structured representations that (1) builds a
semantic map of the scene and (2) performs exploration with a semantic search
policy, to achieve the natural language goal. Our modular method achieves SOTA
performance (24.46 %) with a substantial (8.17 % absolute) gap from previous
work while using less data by eschewing both expert trajectories and low-level
instructions. Leveraging low-level language, however, can further increase our
performance (26.49 %). Our findings suggest that an explicit spatial memory and
a semantic search policy can provide a stronger and more general representation
for state-tracking and guidance, even in the absence of expert trajectories or
low-level instructions.

提出了一种使用语义地图和语义搜索策略实现自然语言目标的模块化方法，不需要使用专家轨迹和低级语言指令，并使用少量数据取得了 SOTA 性能。同时，该研究表明，显式的空间记忆和语义搜索策略可以为状态跟踪和引导提供更强大和更通用的表示方式，即使没有专家轨迹或低级指令。

FILM: 采用模块化方法遵循语言指令

FILM: Following Instructions in Language with Modular Methods

Understanding images and text together is an important aspect of cognition
and building advanced Artificial Intelligence (AI) systems. As a community, we
have achieved good benchmarks over language and vision domains separately,
however joint reasoning is still a challenge for state-of-the-art computer
vision and natural language processing (NLP) systems. We propose a novel task
to derive joint inference about a given image-text modality and compile the
Visuo-Linguistic Question Answering (VLQA) challenge corpus in a question
answering setting. Each dataset item consists of an image and a reading
passage, where questions are designed to combine both visual and textual
information i.e., ignoring either modality would make the question
unanswerable. We first explore the best existing vision-language architectures
to solve VLQA subsets and show that they are unable to reason well. We then
develop a modular method with slightly better baseline performance, but it is
still far behind human performance. We believe that VLQA will be a good
benchmark for reasoning over a visuo-linguistic context. The dataset, code and
leaderboard is available at this https URL

新方法和基础性性能的比较与现有机器学习方法中挑战的问题进行了讨论，最终提出了 Visuo-Linguistic Question Answering 作为计算机视觉和自然语言处理的基准测试集。