We propose a new test to measure a text model's multitask accuracy. The test
covers 57 tasks including elementary mathematics, US history, computer science,
law, and more. To attain high accuracy on this test, models must possess
extensive world knowledge and problem solving ability. We find that while most
recent models have near random-chance accuracy, the very largest GPT-3 model
improves over random chance by almost 20 percentage points on average. However,
on every one of the 57 tasks, the best models still need substantial
improvements before they can reach expert-level accuracy. Models also have
lopsided performance and frequently do not know when they are wrong. Worse,
they still have near-random accuracy on some socially important subjects such
as morality and law. By comprehensively evaluating the breadth and depth of a
model's academic and professional understanding, our test can be used to
analyze models across many tasks and to identify important shortcomings.

论文提出了一种新的测试方法，以测量文本模型的多任务准确性，涵盖了包括数学、历史、计算机科学、法律等 57 项任务，为了达到高准确性，模型必须具备丰富的世界知识和问题解决能力。通过综合评估模型的学术和专业理解的广度和深度，我们的测试可以用于分析许多任务中的模型并确定重要的缺陷。