Generative models have demonstrated human-level proficiency in various
benchmarks across domains like programming, natural sciences, and general
knowledge. Despite these promising results on competitive benchmarks, they
still struggle with seemingly simple problem-solving tasks typically carried
out by elementary-level students. How do state-of-the-art models perform on
standardized tests designed to assess computational thinking and
problem-solving skills at schools? In this paper, we curate a novel benchmark
involving computational thinking tests grounded in elementary visual
programming domains. Our initial results show that state-of-the-art models like
GPT-4o and Llama3 barely match the performance of an average school student. To
further boost the performance of these models, we fine-tune them using a novel
synthetic data generation methodology. The key idea is to develop a
comprehensive dataset using symbolic methods that capture different skill
levels, ranging from recognition of visual elements to multi-choice quizzes to
synthesis-style tasks. We showcase how various aspects of symbolic information
in synthetic data help improve fine-tuned models' performance. We will release
the full implementation and datasets to facilitate further research on
enhancing computational thinking in generative models.

通过使用基于符号方法生成的综合数据集，对生成模型进行微调以提高在计算思维测试中的表现。

在小学可视化编程中对生成模型进行计算思维测试的基准评估

Benchmarking Generative Models on Computational Thinking Tests in  Elementary Visual Programming

Mathematical questioning is crucial for assessing students problem-solving
skills. Since manually creating such questions requires substantial effort,
automatic methods have been explored. Existing state-of-the-art models rely on
fine-tuning strategies and struggle to generate questions that heavily involve
multiple steps of logical and arithmetic reasoning. Meanwhile, large language
models(LLMs) such as ChatGPT have excelled in many NLP tasks involving logical
and arithmetic reasoning. Nonetheless, their applications in generating
educational questions are underutilized, especially in the field of
mathematics. To bridge this gap, we take the first step to conduct an in-depth
analysis of ChatGPT in generating pre-university math questions. Our analysis
is categorized into two main settings: context-aware and context-unaware. In
the context-aware setting, we evaluate ChatGPT on existing math
question-answering benchmarks covering elementary, secondary, and ternary
classes. In the context-unaware setting, we evaluate ChatGPT in generating math
questions for each lesson from pre-university math curriculums that we crawl.
Our crawling results in TopicMath, a comprehensive and novel collection of
pre-university math curriculums collected from 121 math topics and 428 lessons
from elementary, secondary, and tertiary classes. Through this analysis, we aim
to provide insight into the potential of ChatGPT as a math questioner.

ChatGPT 分析了其在生成高中数学问题方面的潜力，通过对各类课程的数学问题进行评估和分析，发现大语言模型在数学问题生成方面具有潜力。