Is aesthetic impact different from beauty? Is visual salience a reflection of
its capacity for effective communication? We present Impressions, a novel
dataset through which to investigate the semiotics of images, and how specific
visual features and design choices can elicit specific emotions, thoughts and
beliefs. We posit that the impactfulness of an image extends beyond formal
definitions of aesthetics, to its success as a communicative act, where style
contributes as much to meaning formation as the subject matter. However, prior
image captioning datasets are not designed to empower state-of-the-art
architectures to model potential human impressions or interpretations of
images. To fill this gap, we design an annotation task heavily inspired by
image analysis techniques in the Visual Arts to collect 1,440 image-caption
pairs and 4,320 unique annotations exploring impact, pragmatic image
description, impressions, and aesthetic design choices. We show that existing
multimodal image captioning and conditional generation models struggle to
simulate plausible human responses to images. However, this dataset
significantly improves their ability to model impressions and aesthetic
evaluations of images through fine-tuning and few-shot adaptation.

通过独特的图像语义学数据集 Impressions，研究图像的符号学以及特定的视觉特征和设计选择如何引发特定的情感、思考和信念，证明了对图像印象的影响力超出了形式美学的描述，而是涉及其作为传达行为的成功，其中风格为意义形成做出的贡献与主题内容一样重要。通过 Fine-tuning 和少样本适应，对现有的多模态图像字幕和条件生成模型进行训练，可以模拟出对图像的合理人类反应，从而有效地模拟图像的印象和美学评价。

印象：理解视觉符号学和美学冲击

Impressions: Understanding Visual Semiotics and Aesthetic Impact

In this paper, we introduce the task of automatically generating text to
describe the differences between two similar images. We collect a new dataset
by crowd-sourcing difference descriptions for pairs of image frames extracted
from video-surveillance footage. Annotators were asked to succinctly describe
all the differences in a short paragraph. As a result, our novel dataset
provides an opportunity to explore models that align language and vision, and
capture visual salience. The dataset may also be a useful benchmark for
coherent multi-sentence generation. We perform a firstpass visual analysis that
exposes clusters of differing pixels as a proxy for object-level differences.
We propose a model that captures visual salience by using a latent variable to
align clusters of differing pixels with output sentences. We find that, for
both single-sentence generation and as well as multi-sentence generation, the
proposed model outperforms the models that use attention alone.

本文介绍了一项任务，即自动生成文本来描述两个相似图像之间的差异。我们通过众包获取了一组新的数据集，并提出了一种模型，该模型使用隐变量来根据像素聚类将不同的像素与输出句子对齐，以捕捉视觉显著性并实现语言和视觉的对准。