In the realm of efficient on-device learning under extreme memory and
computation constraints, a significant gap in successful approaches persists.
Although considerable effort has been devoted to efficient inference, the main
obstacle to efficient learning is the prohibitive cost of backpropagation. The
resources required to compute gradients and update network parameters often
exceed the limits of tightly constrained memory budgets. This paper challenges
conventional wisdom and proposes a series of experiments that reveal the
existence of superior sub-networks. Furthermore, we hint at the potential for
substantial gains through a dynamic neuron selection strategy when fine-tuning
a target task. Our efforts extend to the adaptation of a recent dynamic neuron
selection strategy pioneered by Bragagnolo et al. (NEq), revealing its
effectiveness in the most stringent scenarios. Our experiments demonstrate, in
the average case, the superiority of a NEq-inspired approach over a random
selection. This observation prompts a compelling avenue for further exploration
in the area, highlighting the opportunity to design a new class of algorithms
designed to facilitate parameter update selection. Our findings usher in a new
era of possibilities in the field of on-device learning under extreme
constraints and encourage the pursuit of innovative strategies for efficient,
resource-friendly model fine-tuning.

挤压在极端存储和计算约束下的设备学习所需资源，提出了一种利用动态神经元选择和有效学习的新方法。