The extensive memory footprint of pre-trained language models (PLMs) can
hinder deployment in memory-constrained settings, such as cloud environments or
on-device. PLMs use embedding matrices to represent extensive vocabularies,
forming a large proportion of the model parameters. While previous work towards
parameter-efficient PLM development has considered pruning parameters within
the transformer layers, pruning the embedding matrix as part of fine-tuning or
inference has yet to be explored. We first demonstrate that a significant
proportion of the vocabulary remains unused in these scenarios. We then propose
a simple yet effective approach that leverages this finding to minimize the
memory footprint of the embedding matrix. We show that this approach provides
substantial reductions in memory usage across a wide range of models and tasks.
Notably, our approach maintains equivalent downstream task performance while
allowing a more efficient use of compute resources.

使用预训练语言模型（PLMs）的广泛内存占用可阻碍其在内存受限环境（如云环境或设备上）的部署。本文提出了一种简单而有效的方法来减小嵌入矩阵的内存占用，从而在保持下游任务性能的同时更有效地利用计算资源。