Real-time detection and prediction of extreme weather protect human lives and
infrastructure. Traditional methods rely on numerical threshold setting and
manual interpretation of weather heatmaps with Geographic Information Systems
(GIS), which can be slow and error-prone. Our research redefines Extreme
Weather Events Detection (EWED) by framing it as a Visual Question Answering
(VQA) problem, thereby introducing a more precise and automated solution.
Leveraging Vision-Language Models (VLM) to simultaneously process visual and
textual data, we offer an effective aid to enhance the analysis process of
weather heatmaps. Our initial assessment of general-purpose VLMs (e.g.,
GPT-4-Vision) on EWED revealed poor performance, characterized by low accuracy
and frequent hallucinations due to inadequate color differentiation and
insufficient meteorological knowledge. To address these challenges, we
introduce ClimateIQA, the first meteorological VQA dataset, which includes
8,760 wind gust heatmaps and 254,040 question-answer pairs covering four
question types, both generated from the latest climate reanalysis data. We also
propose Sparse Position and Outline Tracking (SPOT), an innovative technique
that leverages OpenCV and K-Means clustering to capture and depict color
contours in heatmaps, providing ClimateIQA with more accurate color spatial
location information. Finally, we present Climate-Zoo, the first meteorological
VLM collection, which adapts VLMs to meteorological applications using the
ClimateIQA dataset. Experiment results demonstrate that models from Climate-Zoo
substantially outperform state-of-the-art general VLMs, achieving an accuracy
increase from 0% to over 90% in EWED verification. The datasets and models in
this study are publicly available for future climate science research:
this https URL

实时检测和预测极端天气，通过将其作为一个视觉问答问题重新定义，引入了一种更精确、自动化的解决方案。利用视觉语言模型同时处理视觉和文本数据，我们的研究提供了一种有效的辅助分析气象热图过程的方法。通过介绍气候问答数据集 ClimateIQA 和基于 OpenCV 和 K-Means 聚类的 SPOT 技术，我们的模型集合 Climate-Zoo 在极端天气事件检测中取得了显著的性能提升。该研究中的数据集和模型可供未来的气候科学研究使用。