Human feedback plays a critical role in learning and refining reward models
for text-to-image generation, but the optimal form the feedback should take for
learning an accurate reward function has not been conclusively established.
This paper investigates the effectiveness of fine-grained feedback which
captures nuanced distinctions in image quality and prompt-alignment, compared
to traditional coarse-grained feedback (for example, thumbs up/down or ranking
between a set of options). While fine-grained feedback holds promise,
particularly for systems catering to diverse societal preferences, we show that
demonstrating its superiority to coarse-grained feedback is not automatic.
Through experiments on real and synthetic preference data, we surface the
complexities of building effective models due to the interplay of model choice,
feedback type, and the alignment between human judgment and computational
interpretation. We identify key challenges in eliciting and utilizing
fine-grained feedback, prompting a reassessment of its assumed benefits and
practicality. Our findings -- e.g., that fine-grained feedback can lead to
worse models for a fixed budget, in some settings; however, in controlled
settings with known attributes, fine grained rewards can indeed be more helpful
-- call for careful consideration of feedback attributes and potentially beckon
novel modeling approaches to appropriately unlock the potential value of
fine-grained feedback in-the-wild.

通过对真实数据和合成偏好数据的实验，我们发现精细的反馈不一定总是优于粗略的反馈，根据模型选择、反馈类型和人类判断与计算解释的一致性的相互作用存在的复杂性，我们确定了精细反馈获取和利用的关键挑战，这促使我们重新评估其所假设的优势和实用性。我们的发现表明，在某些情况下，对于固定预算，精细的反馈可能导致更差的模型；然而，在已知属性的受控环境中，精细的奖励确实可以更加有帮助，这需要仔细考虑反馈属性并可能呼唤新的建模方法来合理地挖掘精细反馈在实际应用中的潜在价值。