This work presents a study on how to exploit the CLIP embedding space to perform Visual Sentiment Analysis. We experiment with two architectures built on top of the CLIP embedding space, which we denote by CLIP-E. We train the CLIP-E models with WEBEmo, the largest publicly available and manually labeled benchmark for Visual Sentiment Analysis, and perform two sets of experiments. First, we test on WEBEmo and compare the CLIP-E architectures with state-of-the-art (SOTA) models and with CLIP Zero-Shot. Second, we perform cross dataset evaluation, and test the CLIP-E architectures trained with WEBEmo on other Visual Sentiment Analysis benchmarks. Our results show that the CLIP-E approaches outperform SOTA models in WEBEmo fine grained categorization, and they also generalize better when tested on datasets that have not been seen during training. Interestingly, we observed that for the FI dataset, CLIP Zero-Shot produces better accuracies than SOTA models and CLIP-E trained on WEBEmo. These results motivate several questions that we discuss in this paper, such as how we should design new benchmarks and evaluate Visual Sentiment Analysis, and whether we should keep designing tailored Deep Learning models for Visual Sentiment Analysis or focus our efforts on better using the knowledge encoded in large vision-language models such as CLIP for this task.

本研究在探索如何利用CLIP嵌入空间进行视觉情感分析，通过在WEBEmo基准测试上训练CLIP-E模型和进行交叉数据集评估，发现CLIP-E方法在细粒度分类和一些未经训练数据集上的泛化能力均优于现有模型，这引发了关于设计新基准和评估视觉情感分析的问题，并讨论了是否应该继续设计专门的深度学习模型或者更好地利用CLIP等大型视觉-语言模型的知识来解决这一任务。

关于使用视觉语言模型进行视觉情感分析的研究：针对CLIP的一项研究