Resource-constrained devices are increasingly the deployment targets of machine learning applications. Static models, however, do not always suffice for dynamic environments. On-device training of models allows for quick adaptability to new scenarios. With the increasing size of deep neural networks, as noted with the likes of BERT and other natural language processing models, comes increased resource requirements, namely memory, computation, energy, and time. Furthermore, training is far more resource intensive than inference. Resource-constrained on-device learning is thus doubly difficult, especially with large BERT-like models. By reducing the memory usage of fine-tuning, pre-trained BERT models can become efficient enough to fine-tune on resource-constrained devices. We propose Freeze And Reconfigure (FAR), a memory-efficient training regime for BERT-like models that reduces the memory usage of activation maps during fine-tuning by avoiding unnecessary parameter updates. FAR reduces fine-tuning time on the DistilBERT model and CoLA dataset by 30%, and time spent on memory operations by 47%. More broadly, reductions in metric performance on the GLUE and SQuAD datasets are around 1% on average.

本研究提出了一种名为“Freeze And Reconfigure (FAR)”的训练方案，用于在资源受限的设备上以较小的存储空间和时间成本优化BERT模型。实验结果表明，采用FAR训练方案的DistilBERT模型在CoLA数据集上可以将fine-tuning时间降低30％，节省47％的存储操作时间，GLUE和SQuAD等数据集上的度量性能平均降低约1％。

边缘上BERT模型的高效微调