As a natural language assistant, ChatGPT is capable of performing various
tasks, including but not limited to article generation, code completion, and
data analysis. Furthermore, ChatGPT has consistently demonstrated a remarkable
level of accuracy and reliability in terms of content evaluation, exhibiting
the capability of mimicking human preferences. To further explore ChatGPT's
potential in this regard, a study is conducted to assess its ability to rank
content. In order to do so, a test set consisting of prompts is created,
covering a wide range of use cases, and five models are utilized to generate
corresponding responses. ChatGPT is then instructed to rank the responses
generated by these models. The results on the test set show that ChatGPT's
ranking preferences are consistent with human to a certain extent. This
preliminary experimental finding implies that ChatGPT's zero-shot ranking
capability could be used to reduce annotation pressure in a number of ranking
tasks.

本研究评估了 ChatGPT 在内容排序方面的能力，结果表明 ChatGPT 的排序偏好与人类的一定程度上一致。这证明了它具有一定的‘zero-shot ranking’能力，有潜力在许多排序任务中减轻注释压力。