Large language and multimodal models have shown remarkable successes on
various benchmarks focused on specific skills such as general-purpose
programming, natural language understanding, math word problem-solving, and
visual question answering. However, it is unclear how well these models perform
on tasks that require a combination of these skills. In this paper, we curate a
novel program synthesis benchmark based on the XLogoOnline visual programming
environment. The benchmark comprises 85 real-world tasks from the Mini-level of
the XLogoOnline environment, each requiring a combination of different skills
such as spatial planning, basic programming, and logical reasoning. Our
evaluation shows that current state-of-the-art models like GPT-4V and
Llama3-70B struggle to solve these tasks, achieving only 20% and 2.35% success
rates. Next, we develop a fine-tuning pipeline to boost the performance of
models by leveraging a large-scale synthetic training dataset with over 80000
tasks. Moreover, we showcase how emulator-driven feedback can be used to design
a curriculum over training data distribution. We showcase that a fine-tuned
Llama3-8B drastically outperforms GPT-4V and Llama3-70B models, and provide an
in-depth analysis of the models' expertise across different skill dimensions.
We will publicly release the benchmark for future research on program synthesis
in visual programming.

该研究使用大型语言和多模态模型，针对具备空间规划、基础编程和逻辑推理等不同技能要求的 Mini-level 任务，在 XLogoOnline 可视化编程环境下构建了一个新型的程序合成基准。通过使用大规模的合成训练数据集和模拟器驱动的反馈，开发了一个微调流程以提升模型的性能。研究表明，经过微调的 Llama3-8B 模型在不同技能维度上显著优于 GPT-4V 和 Llama3-70B 模型，并提供了对模型在不同技能维度上的专业知识的深入分析。该研究将公开发布该基准，以促进未来在可视化编程的程序合成领域的研究。