Severe convective weather events, such as hail, tornadoes, and thunderstorms,
often occur quickly yet cause significant damage, costing billions of dollars
every year. This highlights the importance of forecasting severe weather
threats hours in advance to better prepare meteorologists and residents in
at-risk areas. Can modern large foundation models perform such forecasting?
Existing weather benchmarks typically focus only on predicting time-series
changes in certain weather parameters (e.g., temperature, moisture) with
text-only features. In this work, we introduce WeatherQA, the first multimodal
dataset designed for machines to reason about complex combinations of weather
parameters (a.k.a., ingredients) and predict severe weather in real-world
scenarios. The dataset includes over 8,000 (multi-images, text) pairs for
diverse severe weather events. Each pair contains rich information crucial for
forecasting -- the images describe the ingredients capturing environmental
instability, surface observations, and radar reflectivity, and the text
contains forecast analyses written by human experts. With WeatherQA, we
evaluate state-of-the-art vision language models , including GPT4, Claude3,
Gemini-1.5, and a fine-tuned Llama3-based VLM, by designing two challenging
tasks: (1) multi-choice QA for predicting affected area and (2) classification
of the development potential of severe convection. These tasks require deep
understanding of domain knowledge (e.g., atmospheric dynamics) and complex
reasoning over multimodal data (e.g., interactions between weather parameters).
We show a substantial gap between the strongest VLM, GPT4o, and human
reasoning. Our comprehensive case study with meteorologists further reveals the
weaknesses of the models, suggesting that better training and data integration
are necessary to bridge this gap. WeatherQA link:
this https URL

多模态数据集 WeatherQA 是用于预测复杂的恶劣天气事件的第一个数据集，使用视觉语言模型评估模型在多选问题和危险密度分类任务上表现较人类推理薄弱，需要更好的训练和数据整合来弥合差距。