We introduce a new challenge to test the STEM skills of neural models. The
problems in the real world often require solutions, combining knowledge from
STEM (science, technology, engineering, and math). Unlike existing datasets,
our dataset requires the understanding of multimodal vision-language
information of STEM. Our dataset features one of the largest and most
comprehensive datasets for the challenge. It includes 448 skills and 1,073,146
questions spanning all STEM subjects. Compared to existing datasets that often
focus on examining expert-level ability, our dataset includes fundamental
skills and questions designed based on the K-12 curriculum. We also add
state-of-the-art foundation models such as CLIP and GPT-3.5-Turbo to our
benchmark. Results show that the recent model advances only help master a very
limited number of lower grade-level skills (2.5% in the third grade) in our
dataset. In fact, these models are still well below (averaging 54.7%) the
performance of elementary students, not to mention near expert-level
performance. To understand and increase the performance on our dataset, we
teach the models on a training split of our dataset. Even though we observe
improved performance, the model performance remains relatively low compared to
average elementary students. To solve STEM problems, we will need novel
algorithmic innovations from the community.

我们介绍了一个新的挑战来测试神经模型的 STEM 技能，我们的数据集涵盖了 STEM 的多模式视觉语言信息，包括了 448 项技能和 1,073,146 个问题，与现有数据集相比，我们的数据集涵盖了从幼儿园到 12 年级课程的基础技能和问题，并添加了 CLIP 和 GPT-3.5-Turbo 等最新的基础模型到我们的基准测试，结果表明，最近的模型进展只有在我们数据集中的一小部分低年级技能（三年级的 2.5%）上有所帮助，事实上，这些模型的性能仍然远低于小学生的平均水平（仅平均 54.7%），更不用说接近专家级性能了，为了提高我们数据集上的模型性能，我们将模型训练在数据集的训练集上，尽管我们观察到性能有所提升，但与普通小学生相比，模型性能仍然相对较低，因此我们需要来自社区的创新算法来解决 STEM 问题。