Multimodal large language models (MLLMs) have shown remarkable capabilities
across a broad range of tasks but their knowledge and abilities in the
geographic and geospatial domains are yet to be explored, despite potential
wide-ranging benefits to navigation, environmental research, urban development,
and disaster response. We conduct a series of experiments exploring various
vision capabilities of MLLMs within these domains, particularly focusing on the
frontier model GPT-4V, and benchmark its performance against open-source
counterparts. Our methodology involves challenging these models with a
small-scale geographic benchmark consisting of a suite of visual tasks, testing
their abilities across a spectrum of complexity. The analysis uncovers not only
where such models excel, including instances where they outperform humans, but
also where they falter, providing a balanced view of their capabilities in the
geographic domain. To enable the comparison and evaluation of future models,
our benchmark will be publicly released.

通过进行一系列实验，我们研究了多模态大型语言模型在地理和地理空间领域的知识和能力，重点关注前沿模型 GPT-4V 的视觉能力，并与开源模型进行性能比较。我们的方法涉及使用一套地理任务的小规模基准测试这些模型，测试它们在不同难度任务上的能力。分析结果揭示了这些模型的优点，包括超过人类的性能，并揭示了它们的不足之处，提供了它们在地理领域能力的全面视角。为了促进未来模型的比较和评估，我们将公开发布我们的基准测试。