The social media landscape of conflict dynamics has grown increasingly
multi-modal. Recent advancements in model architectures such as CLIP have
enabled researchers to begin studying the interplay between the modalities of
text and images in a shared latent space. However, CLIP models fail to handle
situations on social media when modalities present in a post expand above two.
Social media dynamics often require understanding the interplay between not
only text and images, but video as well. In this paper we explore an extension
of the contrastive loss function to allow for any number of modalities, and
demonstrate its usefulness in trimodal spaces on social media. By extending
CLIP into three dimensions we can further aide understanding social media
landscapes where all three modalities are present (an increasingly common
situation). We use a newly collected public data set of Telegram posts
containing all three modalities to train, and then demonstrate the usefulness
of, a trimodal model in two OSINT scenarios: classifying a social media
artifact post as either pro-Russian or pro-Ukrainian and identifying which
account a given artifact originated from. While trimodal CLIP models have been
explored before (though not on social media data), we also display a novel
quadmodal CLIP model. This model can learn the interplay between text, image,
video, and audio. We demonstrate new state-of-the-art baseline results on
retrieval for quadmodel models moving forward.

通过将对比损失函数扩展到任意数量的模态，本文在社交媒体中探讨三模态空间的实用性，并在两个 OSINT 情景中展示了三模态模型的使用价值：将社交媒体文物帖子分类为亲俄罗斯或亲乌克兰，以及识别给定文物的原始账户。此外，本文还展示了一种新型的四模态 CLIP 模型，可以学习文本、图像、视频和音频之间的相互作用。在四模态模型检索中，展示了新的最先进基准结果。