To create culturally inclusive vision-language models (VLMs), the foremost
requirement is developing a test benchmark that can diagnose the models'
ability to respond to questions reflecting cultural elements. This paper
addresses the necessity for such benchmarks, noting that existing research has
relied on human annotators' manual efforts, which impedes diversity and
efficiency. We propose a semi-automated pipeline for constructing cultural VLM
benchmarks to enhance diversity and efficiency. This pipeline leverages
human-VLM collaboration, where VLMs generate questions based on guidelines,
human-annotated examples, and image-wise relevant knowledge, which are then
reviewed by native speakers for quality and cultural relevance. The
effectiveness of our adaptable pipeline is demonstrated through a specific
application: creating a dataset tailored to Korean culture, dubbed K-Viscuit.
The resulting benchmark features two types of questions: Type 1 questions
measure visual recognition abilities, while Type 2 assess fine-grained visual
reasoning skills. This ensures a thorough diagnosis of VLM models across
various aspects. Our evaluation using K-Viscuit revealed that open-source
models notably lag behind proprietary models in understanding Korean culture,
highlighting areas for improvement. We provided diverse analyses of VLM
performance across different cultural aspects. Besides, we explored the
potential of incorporating external knowledge retrieval to enhance the
generation process, suggesting future directions for improving cultural
interpretation ability of VLMs. Our dataset and code will be made publicly
available.

通过半自动化流程，并结合外部知识检索，构建了适应韩国文化的数据集，从而有效评估了文化相关的视觉 - 语言模型，结果显示开源模型在理解韩国文化方面明显落后于专有模型，并提供了各种文化方面的 VLM 性能分析和未来改进方向的建议。