Distributed deep learning has recently been attracting more attention in
remote sensing (RS) applications due to the challenges posed by the increased
amount of open data that are produced daily by Earth observation programs.
However, the high communication costs of sending model updates among multiple
nodes are a significant bottleneck for scalable distributed learning. Gradient
sparsification has been validated as an effective gradient compression (GC)
technique for reducing communication costs and thus accelerating the training
speed. Existing state-of-the-art gradient sparsification methods are mostly
based on the "larger-absolute-more-important" criterion, ignoring the
importance of small gradients, which is generally observed to affect the
performance. Inspired by informative representation of manifold structures from
neighborhood information, we propose a simple yet effective dynamic gradient
compression scheme leveraging neighborhood statistics indicator for RS image
interpretation, termed RS-DGC. We first enhance the interdependence between
gradients by introducing the gradient neighborhood to reduce the effect of
random noise. The key component of RS-DGC is a Neighborhood Statistical
Indicator (NSI), which can quantify the importance of gradients within a
specified neighborhood on each node to sparsify the local gradients before
gradient transmission in each iteration. Further, a layer-wise dynamic
compression scheme is proposed to track the importance changes of each layer in
real time. Extensive downstream tasks validate the superiority of our method in
terms of intelligent interpretation of RS images. For example, we achieve an
accuracy improvement of 0.51% with more than 50 times communication compression
on the NWPU-RESISC45 dataset using VGG-19 network.

近期，分布式深度学习在遥感应用中引起了更多的关注，其中梯度稀疏化被验证为一种有效压缩梯度的方法，以减少通信成本并加快训练速度。通过引入梯度邻域来增强梯度之间的相互依赖，并使用邻域统计指标（NSI）来定义梯度的重要性，我们提出了一种动态梯度压缩方案（RS-DGC）用于遥感图像解释，该方法在智能解释遥感图像方面优于现有方法。

RS-DGC: 通过探索邻域统计量对遥感影像解释的动态梯度压缩

RS-DGC: Exploring Neighborhood Statistics for Dynamic Gradient  Compression on Remote Sensing Image Interpretation

In recent years, remote sensing (RS) vision foundation models such as RingMo
have emerged and achieved excellent performance in various downstream tasks.
However, the high demand for computing resources limits the application of
these models on edge devices. It is necessary to design a more lightweight
foundation model to support on-orbit RS image interpretation. Existing methods
face challenges in achieving lightweight solutions while retaining
generalization in RS image interpretation. This is due to the complex high and
low-frequency spectral components in RS images, which make traditional single
CNN or Vision Transformer methods unsuitable for the task. Therefore, this
paper proposes RingMo-lite, an RS multi-task lightweight network with a
CNN-Transformer hybrid framework, which effectively exploits the
frequency-domain properties of RS to optimize the interpretation process. It is
combined by the Transformer module as a low-pass filter to extract global
features of RS images through a dual-branch structure, and the CNN module as a
stacked high-pass filter to extract fine-grained details effectively.
Furthermore, in the pretraining stage, the designed frequency-domain masked
image modeling (FD-MIM) combines each image patch's high-frequency and
low-frequency characteristics, effectively capturing the latent feature
representation in RS data. As shown in Fig. 1, compared with RingMo, the
proposed RingMo-lite reduces the parameters over 60% in various RS image
interpretation tasks, the average accuracy drops by less than 2% in most of the
scenes and achieves SOTA performance compared to models of the similar size. In
addition, our work will be integrated into the MindSpore computing platform in
the near future.

提出了 RingMo-lite，一种 RS 多任务轻量级网络，具有 CNN-Transformer 混合框架，通过有效利用 RS 的频域特性优化了解释过程。相比于 RingMo，在各种 RS 图像解释任务中，参数减少了 60% 以上，大多数情况下平均准确性下降不到 2%，并在相似规模的模型中实现了 SOTA 性能。