The ability to perform complex tasks from detailed instructions is a key to
many remarkable achievements of our species. As humans, we are not only capable
of performing a wide variety of tasks but also very complex ones that may
entail hundreds or thousands of steps to complete. Large language models and
their more recent multimodal counterparts that integrate textual and visual
inputs have achieved unprecedented success in performing complex tasks. Yet,
most existing benchmarks are largely confined to single-modality inputs (either
text or vision), narrowing the scope of multimodal assessments, particularly
for instruction-following in multimodal contexts. To bridge this gap, we
introduce the instructed-Virtual VISual Decision Making (iWISDM) environment
engineered to generate a limitless array of vision-language tasks of varying
complexity. Using iWISDM, we compiled three distinct benchmarks of instruction
following visual tasks across varying complexity levels and evaluated several
newly developed multimodal models on these benchmarks. Our findings establish
iWISDM as a robust benchmark for assessing the instructional adherence of both
existing and emergent multimodal models and highlight a large gap between these
models' ability to precisely follow instructions with that of humans.

通过 iWISDM 环境，我们编制了三个不同复杂度的指令跟踪视觉任务基准，并评估了几种新开发的多模态模型，结果显示多模态模型的指令遵循能力与人类存在较大差距。