This paper argues in favor of the adoption of annotation practices for
multimodal datasets that recognize and represent the inherently perspectivized
nature of multimodal communication. To support our claim, we present a set of
annotation experiments in which FrameNet annotation is applied to the Multi30k
and the Flickr 30k Entities datasets. We assess the cosine similarity between
the semantic representations derived from the annotation of both pictures and
captions for frames. Our findings indicate that: (i) frame semantic similarity
between captions of the same picture produced in different languages is
sensitive to whether the caption is a translation of another caption or not,
and (ii) picture annotation for semantic frames is sensitive to whether the
image is annotated in presence of a caption or not.

本文主张采用注释实践来认识和代表多模式交流的内在透视性，并通过一系列注释实验介绍了在 Multi30k 和 Flickr 30k Entities 数据集上应用 FrameNet 注释的结果，其发现认为：（一）不同语言中产生的相同图片的标题之间的框架语义相似性对于标题是否为另一标题的翻译敏感，以及（二）图片注释对于语义框架是否在存在标题时进行的 ANNOTATION 是敏感的。