Precise crop yield prediction provides valuable information for agricultural
planning and decision-making processes. However, timely predicting crop yields
remains challenging as crop growth is sensitive to growing season weather
variation and climate change. In this work, we develop a deep learning-based
solution, namely Multi-Modal Spatial-Temporal Vision Transformer (MMST-ViT),
for predicting crop yields at the county level across the United States, by
considering the effects of short-term meteorological variations during the
growing season and the long-term climate change on crops. Specifically, our
MMST-ViT consists of a Multi-Modal Transformer, a Spatial Transformer, and a
Temporal Transformer. The Multi-Modal Transformer leverages both visual remote
sensing data and short-term meteorological data for modeling the effect of
growing season weather variations on crop growth. The Spatial Transformer
learns the high-resolution spatial dependency among counties for accurate
agricultural tracking. The Temporal Transformer captures the long-range
temporal dependency for learning the impact of long-term climate change on
crops. Meanwhile, we also devise a novel multi-modal contrastive learning
technique to pre-train our model without extensive human supervision. Hence,
our MMST-ViT captures the impacts of both short-term weather variations and
long-term climate change on crops by leveraging both satellite images and
meteorological data. We have conducted extensive experiments on over 200
counties in the United States, with the experimental results exhibiting that
our MMST-ViT outperforms its counterparts under three performance metrics of
interest.

通过考虑短期气象变化和长期气候变化对作物的影响，基于多模态时空视觉变换器（MMST-ViT）开发了一个基于深度学习的解决方案，用于预测美国各县的作物产量，并展示其在三个性能指标上优于同类产品。