Vision Language Models (VLMs) have undergone a rapid evolution, giving rise
to significant advancements in the realm of multimodal understanding tasks.
However, the majority of these models are trained and evaluated on
English-centric datasets, leaving a gap in the development and evaluation of
VLMs for other languages, such as Japanese. This gap can be attributed to the
lack of methodologies for constructing VLMs and the absence of benchmarks to
accurately measure their performance. To address this issue, we introduce a
novel benchmark, Japanese Heron-Bench, for evaluating Japanese capabilities of
VLMs. The Japanese Heron-Bench consists of a variety of imagequestion answer
pairs tailored to the Japanese context. Additionally, we present a baseline
Japanese VLM that has been trained with Japanese visual instruction tuning
datasets. Our Heron-Bench reveals the strengths and limitations of the proposed
VLM across various ability dimensions. Furthermore, we clarify the capability
gap between strong closed models like GPT-4V and the baseline model, providing
valuable insights for future research in this domain. We release the benchmark
dataset and training code to facilitate further developments in Japanese VLM
research.

为了解决日语 VLMs（Vision Language Models）的开发和评估中存在的问题，我们引入了一个新的基准测试集，日本 Heron-Bench，用于评估 VLMs 的日语能力。我们还提供了一个基准日语 VLM，它使用了经过日语视觉指导调优的数据集进行训练。通过我们的 Heron-Bench，我们揭示了提出的 VLM 在各种能力维度上的优势和局限性，并明确了像 GPT-4V 这样的强封闭模型与基准模型之间的能力差距，为未来的研究提供了宝贵的见解。为了促进日语 VLM 研究的进一步发展，我们公开发布了基准数据集和训练代码。