As a natural language assistant, ChatGPT is capable of performing various
tasks, including but not limited to article generation, code completion, and
data analysis. Furthermore, ChatGPT has consistently demonstrated a remarkable
level of accuracy and reliability in terms of content evaluation, exhibiting
the capability of mimicking human preferences. To further explore ChatGPT's
potential in this regard, a study is conducted to assess its ability to rank
content. In order to do so, a test set consisting of prompts is created,
covering a wide range of use cases, and five models are utilized to generate
corresponding responses. ChatGPT is then instructed to rank the responses
generated by these models. The results on the test set show that ChatGPT's
ranking preferences are consistent with human to a certain extent. This
preliminary experimental finding implies that ChatGPT's zero-shot ranking
capability could be used to reduce annotation pressure in a number of ranking
tasks.

本研究评估了 ChatGPT 在内容排序方面的能力，结果表明 ChatGPT 的排序偏好与人类的一定程度上一致。这证明了它具有一定的‘zero-shot ranking’能力，有潜力在许多排序任务中减轻注释压力。

探究 ChatGPT 在内容排序方面的能力：一项关于与人类偏好的一致性的初步研究

Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferences

Social media systems rely on user feedback and rating mechanisms for
personalization, ranking, and content filtering. However, when users evaluate
content contributed by fellow users (e.g., by liking a post or voting on a
comment), these evaluations create complex social feedback effects. This paper
investigates how ratings on a piece of content affect its author's future
behavior. By studying four large comment-based news communities, we find that
negative feedback leads to significant behavioral changes that are detrimental
to the community. Not only do authors of negatively-evaluated content
contribute more, but also their future posts are of lower quality, and are
perceived by the community as such. Moreover, these authors are more likely to
subsequently evaluate their fellow users negatively, percolating these effects
through the community. In contrast, positive feedback does not carry similar
effects, and neither encourages rewarded authors to write more, nor improves
the quality of their posts. Interestingly, the authors that receive no feedback
are most likely to leave a community. Furthermore, a structural analysis of the
voter network reveals that evaluations polarize the community the most when
positive and negative votes are equally split.

通过对四个大型基于评论的新闻社区进行研究，本文发现负面反馈会导致明显的行为变化，这种变化不仅促使受到负面评价的作者增加贡献，还会使他们未来的贴文质量降低。而正面反馈则没有相似的影响，不会鼓励受奖励的作者写更多，也不会提高他们文章的质量。此外，没有获得反馈的作者最有可能离开社区。