Large Language Models (LLMs) have achieved impressive performance across
various reasoning tasks. However, even state-of-the-art LLMs such as ChatGPT
are prone to logical errors during their reasoning processes. Existing
solutions, which include deploying task-specific verifiers or voting over
multiple reasoning paths, either require extensive human annotations or fail in
scenarios with inconsistent responses. To address these challenges, we
introduce RankPrompt, a new prompting method that enables LLMs to self-rank
their responses without additional resources. RankPrompt breaks down the
ranking problem into a series of comparisons among diverse responses,
leveraging the inherent capabilities of LLMs to generate chains of comparison
as contextual exemplars. Our experiments across 11 arithmetic and commonsense
reasoning tasks show that RankPrompt significantly enhances the reasoning
performance of ChatGPT and GPT-4, with improvements of up to 13\%. RankPrompt
also excels in LLM-based automatic evaluations for open-ended generation,
aligning with human preferences 74\% of the time in the AlpacaEval set.
Moreover, RankPrompt demonstrates robustness against variations in the
orderings and consistencies of responses.

通过使用 RankPrompt 方法，LLMs 可以自我评级其回答，从而显著提高 ChatGPT 和 GPT-4 的推理表现。