In the realm of efficient on-device learning under extreme memory and
computation constraints, a significant gap in successful approaches persists.
Although considerable effort has been devoted to efficient inference, the main
obstacle to efficient learning is the prohibitive cost of backpropagation. The
resources required to compute gradients and update network parameters often
exceed the limits of tightly constrained memory budgets. This paper challenges
conventional wisdom and proposes a series of experiments that reveal the
existence of superior sub-networks. Furthermore, we hint at the potential for
substantial gains through a dynamic neuron selection strategy when fine-tuning
a target task. Our efforts extend to the adaptation of a recent dynamic neuron
selection strategy pioneered by Bragagnolo et al. (NEq), revealing its
effectiveness in the most stringent scenarios. Our experiments demonstrate, in
the average case, the superiority of a NEq-inspired approach over a random
selection. This observation prompts a compelling avenue for further exploration
in the area, highlighting the opportunity to design a new class of algorithms
designed to facilitate parameter update selection. Our findings usher in a new
era of possibilities in the field of on-device learning under extreme
constraints and encourage the pursuit of innovative strategies for efficient,
resource-friendly model fine-tuning.

挤压在极端存储和计算约束下的设备学习所需资源，提出了一种利用动态神经元选择和有效学习的新方法。

边缘上的设备学习：在预算约束下选择要更新的神经元的方法

Towards On-device Learning on the Edge: Ways to Select Neurons to Update  under a Budget Constraint

We propose the width-resolution mutual learning method (MutualNet) to train a
network that is executable at dynamic resource constraints to achieve adaptive
accuracy-efficiency trade-offs at runtime. Our method trains a cohort of
sub-networks with different widths using different input resolutions to
mutually learn multi-scale representations for each sub-network. It achieves
consistently better ImageNet top-1 accuracy over the state-of-the-art adaptive
network US-Net under different computation constraints, and outperforms the
best compound scaled MobileNet in EfficientNet by 1.5%. The superiority of our
method is also validated on COCO object detection and instance segmentation as
well as transfer learning. Surprisingly, the training strategy of MutualNet can
also boost the performance of a single network, which substantially outperforms
the powerful AutoAugmentation in both efficiency (GPU search hours: 15000 vs.
0) and accuracy (ImageNet: 77.6% vs. 78.6%). Code is available at
https://github.com/taoyang1122/MutualNet.

提出了宽度 - 分辨率互学习方法 MutualNet，通过训练不同宽度的多个子网络来实现自适应准确性和效率之间的权衡，最终在不同计算约束下，通过互学习获得更好的 ImageNet top-1 精度。