Modern vision models are trained on very large noisy datasets. While these
models acquire strong capabilities, they may not follow the user's intent to
output the desired results in certain aspects, e.g., visual aesthetic,
preferred style, and responsibility. In this paper, we target the realm of
visual aesthetics and aim to align vision models with human aesthetic standards
in a retrieval system. Advanced retrieval systems usually adopt a cascade of
aesthetic models as re-rankers or filters, which are limited to low-level
features like saturation and perform poorly when stylistic, cultural or
knowledge contexts are involved. We find that utilizing the reasoning ability
of large language models (LLMs) to rephrase the search query and extend the
aesthetic expectations can make up for this shortcoming. Based on the above
findings, we propose a preference-based reinforcement learning method that
fine-tunes the vision models to distill the knowledge from both LLMs reasoning
and the aesthetic models to better align the vision models with human
aesthetics. Meanwhile, with rare benchmarks designed for evaluating retrieval
systems, we leverage large multi-modality model (LMM) to evaluate the aesthetic
performance with their strong abilities. As aesthetic assessment is one of the
most subjective tasks, to validate the robustness of LMM, we further propose a
novel dataset named HPIR to benchmark the alignment with human aesthetics.
Experiments demonstrate that our method significantly enhances the aesthetic
behaviors of the vision models, under several metrics. We believe the proposed
algorithm can be a general practice for aligning vision models with human
values.

利用大型语言模型 (LLM) 的推理能力和美学模型，在一个检索系统中，提出了一种基于偏好的强化学习方法，用于调整视觉模型以更好地与人类美学对齐并提升视觉模型的美学行为。通过使用大型多模型 (LMM) 评估美学表现，并引入名为 HPIR 的新数据集来验证 LMM 的鲁棒性，实验证明了该方法在多个指标下显著提高了视觉模型的美学行为。我们相信该算法可以成为与人类价值观对齐的视觉模型的一种通用方法。

与人类审美观在检索中对齐的视觉模型：基准和算法

Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks  and Algorithms

How do we determine whether two or more clothing items are compatible or
visually appealing? Part of the answer lies in understanding of visual
aesthetics, and is biased by personal preferences shaped by social attitudes,
time, and place. In this work we propose a method that predicts compatibility
between two items based on their visual features, as well as their context. We
define context as the products that are known to be compatible with each of
these item. Our model is in contrast to other metric learning approaches that
rely on pairwise comparisons between item features alone. We address the
compatibility prediction problem using a graph neural network that learns to
generate product embeddings conditioned on their context. We present results
for two prediction tasks (fill in the blank and outfit compatibility) tested on
two fashion datasets Polyvore and Fashion-Gen, and on a subset of the Amazon
dataset; we achieve state of the art results when using context information and
show how test performance improves as more context is used.

本文介绍一种基于图神经网络的方法，通过产品嵌入向量预测衣物单品或搭配的视觉配合性，相较于其他基于成对比较的模型，我们的模型涵盖更丰富的背景信息取得了业界领先的效果。

上下文感知的视觉相容性预测

Context-Aware Visual Compatibility Prediction

Thumbnails play such an important role in online videos. As the most
representative snapshot, they capture the essence of a video and provide the
first impression to the viewers; ultimately, a great thumbnail makes a video
more attractive to click and watch. We present an automatic thumbnail selection
system that exploits two important characteristics commonly associated with
meaningful and attractive thumbnails: high relevance to video content and
superior visual aesthetic quality. Our system selects attractive thumbnails by
analyzing various visual quality and aesthetic metrics of video frames, and
performs a clustering analysis to determine the relevance to video content,
thus making the resulting thumbnails more representative of the video. On the
task of predicting thumbnails chosen by professional video editors, we
demonstrate the effectiveness of our system against six baseline methods, using
a real-world dataset of 1,118 videos collected from Yahoo Screen. In addition,
we study what makes a frame a good thumbnail by analyzing the statistical
relationship between thumbnail frames and non-thumbnail frames in terms of
various image quality features. Our study suggests that the selection of a good
thumbnail is highly correlated with objective visual quality metrics, such as
the frame texture and sharpness, implying the possibility of building an
automatic thumbnail selection system based on visual aesthetics.

我们提出了一种自动缩略图选择系统，该系统利用与有意义且有吸引力的缩略图常见相关的两个重要特征：与视频内容的高相关性和卓越的视觉美学质量。该系统通过分析视频帧的各种视觉质量和美学度量，执行聚类分析以确定与视频内容的相关性，从而使产生的缩略图更具代表性。我们使用从 Yahoo Screen 收集的 1,118 个视频的实际数据集针对六种基准方法展示了我们的系统的有效性，以预测专业视频编辑选择的缩略图任务。此外，我们通过分析缩略帧和非缩略帧的各种图像质量特征的统计关系，研究了什么使帧成为良好的缩略图。我们的研究表明，选择好的缩略图与客观视觉质量度量高度相关，如帧纹理和锐利度，暗示了基于视觉美学构建自动缩略图选择系统的可能性。