On-device continual learning (CL) requires the co-optimization of model
accuracy and resource efficiency to be practical. This is extremely challenging
because it must preserve accuracy while learning new tasks with continuously
drifting data and maintain both high energy and memory efficiency to be
deployable on real-world devices. Typically, a CL method leverages one of two
types of backbone networks: CNN or ViT. It is commonly believed that CNN-based
CL excels in resource efficiency, whereas ViT-based CL is superior in model
performance, making each option attractive only for a single aspect. In this
paper, we revisit this comparison while embracing powerful pre-trained ViT
models of various sizes, including ViT-Ti (5.8M parameters). Our detailed
analysis reveals that many practical options exist today for making ViT-based
methods more suitable for on-device CL, even when accuracy, energy, and memory
are all considered. To further expand this impact, we introduce REP, which
improves resource efficiency specifically targeting prompt-based rehearsal-free
methods. Our key focus is on avoiding catastrophic trade-offs with accuracy
while trimming computational and memory costs throughout the training process.
We achieve this by exploiting swift prompt selection that enhances input data
using a carefully provisioned model, and by developing two novel
algorithms-adaptive token merging (AToM) and adaptive layer dropping (ALD)-that
optimize the prompt updating stage. In particular, AToM and ALD perform
selective skipping across the data and model-layer dimensions without
compromising task-specific features in vision transformer models. Extensive
experiments on three image classification datasets validate REP's superior
resource efficiency over current state-of-the-art methods.

利用先进的预训练 ViT 模型，本研究通过 REP 方法提高基于 ViT 的 on-device continual learning 的资源利用效率，避免牺牲准确性的情况下降低计算和内存成本。