A well-executed graphic design typically achieves harmony in two levels, from
the fine-grained design elements (color, font and layout) to the overall
design. This complexity makes the comprehension of graphic design challenging,
for it needs the capability to both recognize the design elements and
understand the design. With the rapid development of Multimodal Large Language
Models (MLLMs), we establish the DesignProbe, a benchmark to investigate the
capability of MLLMs in design. Our benchmark includes eight tasks in total,
across both the fine-grained element level and the overall design level. At
design element level, we consider both the attribute recognition and semantic
understanding tasks. At overall design level, we include style and metaphor. 9
MLLMs are tested and we apply GPT-4 as evaluator. Besides, further experiments
indicates that refining prompts can enhance the performance of MLLMs. We first
rewrite the prompts by different LLMs and found increased performances appear
in those who self-refined by their own LLMs. We then add extra task knowledge
in two different ways (text descriptions and image examples), finding that
adding images boost much more performance over texts.

多模式大型语言模型 (MLLMs) 在设计方面的能力是 DesignProbe 基准测试旨在研究的重点。通过对两个级别的设计元素和整体设计进行八个任务的测试，我们发现改进提示可以提高 MLLMs 的性能，并且添加图像比添加文本更能提升性能。