This paper introduces WeatherFormer, a transformer encoder-based model
designed to learn robust weather features from minimal observations. It
addresses the challenge of modeling complex weather dynamics from small
datasets, a bottleneck for many prediction tasks in agriculture, epidemiology,
and climate science. WeatherFormer was pretrained on a large pretraining
dataset comprised of 39 years of satellite measurements across the Americas.
With a novel pretraining task and fine-tuning, WeatherFormer achieves
state-of-the-art performance in county-level soybean yield prediction and
influenza forecasting. Technical innovations include a unique spatiotemporal
encoding that captures geographical, annual, and seasonal variations, adapting
the transformer architecture to continuous weather data, and a pretraining
strategy to learn representations that are robust to missing weather features.
This paper for the first time demonstrates the effectiveness of pretraining
large transformer encoder models for weather-dependent applications across
multiple domains.

本文介绍了 WeatherFormer，一种基于 Transformer 编码器的模型，旨在从少量观测数据中学习稳健的天气特征。它解决了在农业、流行病学和气候科学中许多预测任务中建模复杂天气动态的挑战。WeatherFormer 在大型预训练数据集上进行了预训练，包括 39 年在美洲范围内的卫星测量数据。通过独特的预训练任务和微调，WeatherFormer 在县级大豆产量预测和流感预测方面实现了最先进的性能。技术创新包括捕捉地理、年度和季节性变化的独特时空编码，将 Transformer 架构适应连续天气数据，以及一种能够学习对缺失天气特征具有稳健性的表示的预训练策略。本文首次证明了使用大型 Transformer 编码器模型在多个领域的天气相关应用中进行预训练的有效性。

WeatherFormer：一个基于预训练编码器模型的小数据集天气表示学习方法

WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather  Representations from Small Datasets

The prediction of solar power generation is a challenging task due to its
dependence on climatic characteristics that exhibit spatial and temporal
variability. The performance of a prediction model may vary across different
places due to changes in data distribution, resulting in a model that works
well in one region but not in others. Furthermore, as a consequence of global
warming, there is a notable acceleration in the alteration of weather patterns
on an annual basis. This phenomenon introduces the potential for diminished
efficacy of existing models, even within the same geographical region, as time
progresses. In this paper, a domain adaptive deep learning-based framework is
proposed to estimate solar power generation using weather features that can
solve the aforementioned challenges. A feed-forward deep convolutional network
model is trained for a known location dataset in a supervised manner and
utilized to predict the solar power of an unknown location later. This adaptive
data-driven approach exhibits notable advantages in terms of computing speed,
storage efficiency, and its ability to improve outcomes in scenarios where
state-of-the-art non-adaptive methods fail. Our method has shown an improvement
of $10.47 \%$, $7.44 \%$, $5.11\%$ in solar power prediction accuracy compared
to best performing non-adaptive method for California (CA), Florida (FL) and
New York (NY), respectively.

提出了一个基于领域自适应深度学习的框架来估计太阳能发电量，该框架利用天气特征来解决太阳能发电预测中的挑战。该方法在计算速度、存储效率和改善预测准确性方面在加利福尼亚（CA）、佛罗里达（FL）和纽约（NY）等地的太阳能发电预测中显示出明显优势。