Modern vision models are trained on very large noisy datasets. While these
models acquire strong capabilities, they may not follow the user's intent to
output the desired results in certain aspects, e.g., visual aesthetic,
preferred style, and responsibility. In this paper, we target the realm of
visual aesthetics and aim to align vision models with human aesthetic standards
in a retrieval system. Advanced retrieval systems usually adopt a cascade of
aesthetic models as re-rankers or filters, which are limited to low-level
features like saturation and perform poorly when stylistic, cultural or
knowledge contexts are involved. We find that utilizing the reasoning ability
of large language models (LLMs) to rephrase the search query and extend the
aesthetic expectations can make up for this shortcoming. Based on the above
findings, we propose a preference-based reinforcement learning method that
fine-tunes the vision models to distill the knowledge from both LLMs reasoning
and the aesthetic models to better align the vision models with human
aesthetics. Meanwhile, with rare benchmarks designed for evaluating retrieval
systems, we leverage large multi-modality model (LMM) to evaluate the aesthetic
performance with their strong abilities. As aesthetic assessment is one of the
most subjective tasks, to validate the robustness of LMM, we further propose a
novel dataset named HPIR to benchmark the alignment with human aesthetics.
Experiments demonstrate that our method significantly enhances the aesthetic
behaviors of the vision models, under several metrics. We believe the proposed
algorithm can be a general practice for aligning vision models with human
values.

利用大型语言模型 (LLM) 的推理能力和美学模型，在一个检索系统中，提出了一种基于偏好的强化学习方法，用于调整视觉模型以更好地与人类美学对齐并提升视觉模型的美学行为。通过使用大型多模型 (LMM) 评估美学表现，并引入名为 HPIR 的新数据集来验证 LMM 的鲁棒性，实验证明了该方法在多个指标下显著提高了视觉模型的美学行为。我们相信该算法可以成为与人类价值观对齐的视觉模型的一种通用方法。