In this report, we pose the following question: Who is the most intelligent
AI model to date, as measured by the OlympicArena (an Olympic-level,
multi-discipline, multi-modal benchmark for superintelligent AI)? We
specifically focus on the most recently released models: Claude-3.5-Sonnet,
Gemini-1.5-Pro, and GPT-4o. For the first time, we propose using an Olympic
medal Table approach to rank AI models based on their comprehensive performance
across various disciplines. Empirical results reveal: (1) Claude-3.5-Sonnet
shows highly competitive overall performance over GPT-4o, even surpassing
GPT-4o on a few subjects (i.e., Physics, Chemistry, and Biology). (2)
Gemini-1.5-Pro and GPT-4V are ranked consecutively just behind GPT-4o and
Claude-3.5-Sonnet, but with a clear performance gap between them. (3) The
performance of AI models from the open-source community significantly lags
behind these proprietary models. (4) The performance of these models on this
benchmark has been less than satisfactory, indicating that we still have a long
way to go before achieving superintelligence. We remain committed to
continuously tracking and evaluating the performance of the latest powerful
models on this benchmark (available at
this https URL).

谁是迄今为止最智能的人工智能模型？我们使用奥林匹克竞技场（一个超智能人工智能的奥林匹克级多学科多模式基准）进行评估，并提出使用奥林匹克奖牌榜来排名人工智能模型。实证结果表明：Claude-3.5-Sonnet 在各个学科具有竞争力的综合表现优于 GPT-4o，超过 GPT-4o 在物理、化学和生物学等学科。Gemini-1.5-Pro 和 GPT-4V 排名次于 GPT-4o 和 Claude-3.5-Sonnet，但它们之间存在明显的性能差距。开源社区的人工智能模型性能明显落后于专有模型。这些模型在这个基准上的表现不尽人意，表明我们在实现超智能之前还有很长的路要走。我们致力于持续追踪和评估最新强大模型在这个基准上的表现。

奥林匹克竞技场奖牌排名：迄今最聪明的人工智能是谁？

OlympicArena Medal Ranks: Who Is the Most Intelligent AI So Far?

The evolution of Artificial Intelligence (AI) has been significantly
accelerated by advancements in Large Language Models (LLMs) and Large
Multimodal Models (LMMs), gradually showcasing potential cognitive reasoning
abilities in problem-solving and scientific discovery (i.e., AI4Science) once
exclusive to human intellect. To comprehensively evaluate current models'
performance in cognitive reasoning abilities, we introduce OlympicArena, which
includes 11,163 bilingual problems across both text-only and interleaved
text-image modalities. These challenges encompass a wide range of disciplines
spanning seven fields and 62 international Olympic competitions, rigorously
examined for data leakage. We argue that the challenges in Olympic competition
problems are ideal for evaluating AI's cognitive reasoning due to their
complexity and interdisciplinary nature, which are essential for tackling
complex scientific challenges and facilitating discoveries. Beyond evaluating
performance across various disciplines using answer-only criteria, we conduct
detailed experiments and analyses from multiple perspectives. We delve into the
models' cognitive reasoning abilities, their performance across different
modalities, and their outcomes in process-level evaluations, which are vital
for tasks requiring complex reasoning with lengthy solutions. Our extensive
evaluations reveal that even advanced models like GPT-4o only achieve a 39.97%
overall accuracy, illustrating current AI limitations in complex reasoning and
multimodal integration. Through the OlympicArena, we aim to advance AI towards
superintelligence, equipping it to address more complex challenges in science
and beyond. We also provide a comprehensive set of resources to support AI
research, including a benchmark dataset, an open-source annotation platform, a
detailed evaluation tool, and a leaderboard with automatic submission features.

人工智能的进化已经显著加速，主要得益于大型语言模型和大型多模态模型的进展，在问题解决和科学发现方面逐渐展现出与人类智力相当的潜在认知推理能力（即 AI4Science）。为了全面评估当前模型在认知推理能力方面的表现，我们引入了奥林匹克竞技场（OlympicArena），其中包含了 11,163 个双语问题，涵盖了文本和图像两种模态。这些挑战囊括了七个领域和 62 个国际奥林匹克竞赛的各种学科，经过严谨的数据泄露检测。我们认为，奥林匹克竞赛问题中的挑战非常适合评估 AI 的认知推理能力，因为这些问题具有复杂性和跨学科性质，对于解决复杂科学难题和促进发现非常重要。通过答案评估各学科中的表现之外，我们从多个角度进行了详细的实验和分析，深入探讨了模型的认知推理能力，它们在不同模态下的表现以及在解决长篇解答复杂推理任务中的结果。我们广泛的评估显示，即使是像 GPT-4o 这样的先进模型也只能达到 39.97% 的整体准确率，这说明当前人工智能在复杂推理和多模态整合方面存在局限性。通过奥林匹克竞技场，我们旨在推进人工智能走向超级智能，使其能够应对更复杂的科学和其他挑战。我们还提供了一套全面的资源来支持人工智能研究，包括基准数据集、开源注释平台、详细评估工具和带有自动提交功能的排行榜。