Prior work on ideology prediction has largely focused on single modalities,
i.e., text or images. In this work, we introduce the task of multimodal
ideology prediction, where a model predicts binary or five-point scale
ideological leanings, given a text-image pair with political content. We first
collect five new large-scale datasets with English documents and images along
with their ideological leanings, covering news articles from a wide range of US
mainstream media and social media posts from Reddit and Twitter. We conduct
in-depth analyses of news articles and reveal differences in image content and
usage across the political spectrum. Furthermore, we perform extensive
experiments and ablation studies, demonstrating the effectiveness of targeted
pretraining objectives on different model components. Our best-performing
model, a late-fusion architecture pretrained with a triplet objective over
multimodal content, outperforms the state-of-the-art text-only model by almost
4% and a strong multimodal baseline with no pretraining by over 3%.

本文介绍了以文本和图像为输入的多模态意识形态预测任务，提出了使用针对模型组成部分的预训练目标的效果，建立五个包含政治内容的大型数据集，并通过实验和分析实现了在文本分析中更好的性能。