Computing author intent from multimodal data like Instagram posts requires
modeling a complex relationship between text and image. For example, a caption
might evoke an ironic contrast with the image, so neither caption nor image is
a mere transcript of the other. Instead they combine -- via what has been
called meaning multiplication -- to create a new meaning that has a more
complex relation to the literal meanings of text and image. Here we introduce a
multimodal dataset of 1299 Instagram posts labeled for three orthogonal
taxonomies: the authorial intent behind the image-caption pair, the contextual
relationship between the literal meanings of the image and caption, and the
semiotic relationship between the signified meanings of the image and caption.
We build a baseline deep multimodal classifier to validate the taxonomy,
showing that employing both text and image improves intent detection by 9.6%
compared to using only the image modality, demonstrating the commonality of
non-intersective meaning multiplication. The gain with multimodality is
greatest when the image and caption diverge semiotically. Our dataset offers a
new resource for the study of the rich meanings that result from pairing text
and image.

通过模型化文本和图像之间的复杂关系，可以计算出像 Instagram 帖子中的作者意图。本文介绍了一个包含 1299 个 Instagram 帖子的多模态数据集，以确定帖子图像标题对之间的意图、上下文关系和符号关系。同时，构建了一个多模态分类器，通过同时使用文本和图像，相较于只使用图像模态，将意图检测的准确性提高了 9.6％。这些结果表明，非相交意义乘法在计算作者意图时具有普遍性。该数据集为研究文本和图像组合产生的丰富含义提供了新的资源。