Assessing the capabilities of large multimodal models (LMMs) often requires
the creation of ad-hoc evaluations. Currently, building new benchmarks requires
tremendous amounts of manual work for each specific analysis. This makes the
evaluation process tedious and costly. In this paper, we present APEx,
Automatic Programming of Experiments, the first framework for automatic
benchmarking of LMMs. Given a research question expressed in natural language,
APEx leverages a large language model (LLM) and a library of pre-specified
tools to generate a set of experiments for the model at hand, and progressively
compile a scientific report. The report drives the testing procedure: based on
the current status of the investigation, APEx chooses which experiments to
perform and whether the results are sufficient to draw conclusions. Finally,
the LLM refines the report, presenting the results to the user in natural
language. Thanks to its modularity, our framework is flexible and extensible as
new tools become available. Empirically, APEx reproduces the findings of
existing studies while allowing for arbitrary analyses and hypothesis testing.

自动化实验设计的框架 APEx 可以借助大型语言模型和预先指定的工具库生成一组实验，并逐步编制科学报告，以驱动测试过程和生成自然语言的结果。