Large language models (LLMs) have demonstrated impressive reasoning
capabilities, yet there is ongoing debate about these abilities and the
potential data contamination problem recently. This paper aims to evaluate the
reasoning capacities of LLMs, specifically in solving recent competition-level
programming problems in Codeforces, which are expert-crafted and unique,
requiring deep understanding and robust reasoning skills. We first provide a
comprehensive evaluation of GPT-4's peiceived zero-shot performance on this
task, considering various aspects such as problems' release time, difficulties,
and types of errors encountered. Surprisingly, the peiceived performance of
GPT-4 has experienced a cliff like decline in problems after September 2021
consistently across all the difficulties and types of problems, which shows the
potential data contamination, as well as the challenges for any existing LLM to
solve unseen complex reasoning problems. We further explore various approaches
such as fine-tuning, Chain-of-Thought prompting and problem description
simplification, unfortunately none of them is able to consistently mitigate the
challenges. Through our work, we emphasis the importance of this excellent data
source for assessing the genuine reasoning capabilities of LLMs, and foster the
development of LLMs with stronger reasoning abilities and better generalization
in the future.

评估大语言模型在 Codeforces 编程问题上的推理能力，发现了潜在的数据污染问题，并探索了多种方法来解决这些挑战，强调了评估 LLMs 真正推理能力的重要性和未来更强大推理能力和更好泛化性能的发展。

竞赛级问题对 LLMs 的有效性评估

Competition-Level Problems Are Effective Evaluators of LLMs

In recent days, the number of technology enthusiasts is increasing day by day
with the prevalence of technological products and easy access to the internet.
Similarly, the amount of people working behind this rapid development is rising
tremendously. Computer programmers consist of a large portion of those
tech-savvy people. Codeforces, an online programming and contest hosting
platform used by many competitive programmers worldwide. It is regarded as one
of the most standardized platforms for practicing programming problems and
participate in programming contests. In this research, we propose a framework
that predicts the performance of any particular contestant in the upcoming
competitions as well as predicts the rating after that contest based on their
practice and the performance of their previous contests.

本研究提出了一个框架，基于选手之前参加比赛的表现和练习情况，预测参加 Codeforces 平台竞赛的选手在未来比赛中的表现和评级。