Accurate in-season crop type classification is crucial for the crop
production estimation and monitoring of agricultural parcels. However, the
complexity of the plant growth patterns and their spatio-temporal variability
present significant challenges. While current deep learning-based methods show
promise in crop type classification from single- and multi-modal time series,
most existing methods rely on a single modality, such as satellite optical
remote sensing data or crop rotation patterns. We propose a novel approach to
fuse multimodal information into a model for improved accuracy and robustness
across multiple years and countries. The approach relies on three modalities
used: remote sensing time series from Sentinel-2 and Landsat 8 observations,
parcel crop rotation and local crop distribution. To evaluate our approach, we
release a new annotated dataset of 7.4 million agricultural parcels in France
and Netherlands. We associate each parcel with time-series of surface
reflectance (Red and NIR) and biophysical variables (LAI, FAPAR). Additionally,
we propose a new approach to automatically aggregate crop types into a
hierarchical class structure for meaningful model evaluation and a novel
data-augmentation technique for early-season classification. Performance of the
multimodal approach was assessed at different aggregation level in the semantic
domain spanning from 151 to 8 crop types or groups. It resulted in accuracy
ranging from 91\% to 95\% for NL dataset and from 85\% to 89\% for FR dataset.
Pre-training on a dataset improves domain adaptation between countries,
allowing for cross-domain zero-shot learning, and robustness of the
performances in a few-shot setting from France to Netherlands. Our proposed
approach outperforms comparable methods by enabling learning methods to use the
often overlooked spatio-temporal context of parcels, resulting in increased
preci...

提出了一种融合多模态信息的方法，用于改进作物类型分类的准确性和稳健性，并针对 740 万个农业地块发布了新的注释数据集。新方法在评估中表现更好，可用于跨领域零样本学习和从法国到荷兰的少样本设定的稳健性。