This paper develops small vision language models to understand visual art,
which, given an art work, aims to identify its emotion category and explain
this prediction with natural language. While small models are computationally
efficient, their capacity is much limited compared with large models. To break
this trade-off, this paper builds a small emotional vision language model
(SEVLM) by emotion modeling and input-output feature alignment. On the one
hand, based on valence-arousal-dominance (VAD) knowledge annotated by
psychology experts, we introduce and fuse emotional features derived through
VAD dictionary and a VAD head to align VAD vectors of predicted emotion
explanation and the ground truth. This allows the vision language model to
better understand and generate emotional texts, compared with using traditional
text embeddings alone. On the other hand, we design a contrastive head to pull
close embeddings of the image, its emotion class, and explanation, which aligns
model outputs and inputs. On two public affective explanation datasets, we show
that the proposed techniques consistently improve the visual art understanding
performance of baseline SEVLMs. Importantly, the proposed model can be trained
and evaluated on a single RTX 2080 Ti while exhibiting very strong performance:
it not only outperforms the state-of-the-art small models but is also
competitive compared with LLaVA 7B after fine-tuning and GPT4(V).

该研究开发了小型视觉语言模型，旨在通过情感建模和输入输出特征对齐，识别视觉艺术作品的情感类别并解释预测结果，实验证明该模型在理解视觉艺术方面的性能明显优于基准模型。