3D content creation from text prompts has shown remarkable success recently.
However, current text-to-3D methods often generate 3D results that do not align
well with human preferences. In this paper, we present a comprehensive
framework, coined DreamReward, to learn and improve text-to-3D models from
human preference feedback. To begin with, we collect 25k expert comparisons
based on a systematic annotation pipeline including rating and ranking. Then,
we build Reward3D -- the first general-purpose text-to-3D human preference
reward model to effectively encode human preferences. Building upon the 3D
reward model, we finally perform theoretical analysis and present the Reward3D
Feedback Learning (DreamFL), a direct tuning algorithm to optimize the
multi-view diffusion models with a redefined scorer. Grounded by theoretical
proof and extensive experiment comparisons, our DreamReward successfully
generates high-fidelity and 3D consistent results with significant boosts in
prompt alignment with human intention. Our results demonstrate the great
potential for learning from human feedback to improve text-to-3D models.

从人类偏好反馈中学习和改进文本生成 3D 模型的综合框架，通过理论分析和广泛实验比较，我们的 DreamReward 成功生成高保真和与人类意图一致的 3D 结果，展示了从人类反馈中改进文本生成 3D 模型的巨大潜力。

DreamReward: 人类偏好的文本生成三维图形

DreamReward: Text-to-3D Generation with Human Preference

Researchers have proposed to use data of human preference feedback to
fine-tune text-to-image generative models. However, the scalability of human
feedback collection has been limited by its reliance on manual annotation.
Therefore, we develop and test a method to automatically annotate user
preferences from their spontaneous facial expression reaction to the generated
images. We collect a dataset of Facial Expression Reaction to Generated Images
(FERGI) and show that the activations of multiple facial action units (AUs) are
highly correlated with user evaluations of the generated images. Specifically,
AU4 (brow lowerer) is most consistently reflective of negative evaluations of
the generated image. This can be useful in two ways. Firstly, we can
automatically annotate user preferences between image pairs with substantial
difference in AU4 responses to them with an accuracy significantly
outperforming state-of-the-art scoring models. Secondly, directly integrating
the AU4 responses with the scoring models improves their consistency with human
preferences. Additionally, the AU4 response best reflects the user's evaluation
of the image fidelity, making it complementary to the state-of-the-art scoring
models, which are generally better at reflecting image-text alignment. Finally,
this method of automatic annotation with facial expression analysis can be
potentially generalized to other generation tasks. The code is available at
this https URL, and the dataset is also available at
the same link for research purposes.

利用面部表情分析进行自动注释，以解决人类偏好反馈数据的收集速度问题，并展示了多个面部动作单元与用户对生成图像的评估高度相关。面部动作单元 AU4 最能反映图像的质量评估，可以用于自动注释用户对图像偏好，并与评分模型结合以提升一致性，同时也能推广至其他生成任务。