Datasets that capture the connection between vision, language, and affection
are limited, causing a lack of understanding of the emotional aspect of human
intelligence. As a step in this direction, the ArtEmis dataset was recently
introduced as a large-scale dataset of emotional reactions to images along with
language explanations of these chosen emotions. We observed a significant
emotional bias towards instance-rich emotions, making trained neural speakers
less accurate in describing under-represented emotions. We show that collecting
new data, in the same way, is not effective in mitigating this emotional bias.
To remedy this problem, we propose a contrastive data collection approach to
balance ArtEmis with a new complementary dataset such that a pair of similar
images have contrasting emotions (one positive and one negative). We collected
260,533 instances using the proposed method, we combine them with ArtEmis,
creating a second iteration of the dataset. The new combined dataset, dubbed
ArtEmis v2.0, has a balanced distribution of emotions with explanations
revealing more fine details in the associated painting. Our experiments show
that neural speakers trained on the new dataset improve CIDEr and METEOR
evaluation metrics by 20% and 7%, respectively, compared to the biased dataset.
Finally, we also show that the performance per emotion of neural speakers is
improved across all the emotion categories, significantly on under-represented
emotions. The collected dataset and code are available at
this https URL.

本研究介绍了 ArtEmis 数据集及其第二版，提出使用对比数据采集方法解决数据集中情感偏向问题，并表明在新数据集上训练的编码 - 解码模型性能在所有情感类别上显著提高，并带来 20% 和 7% 的 CIDEr 和 METEOR 评估指标改善，尤其是在情感匮乏的场景下。