Traditional neural word embeddings are usually dependent on a richer
diversity of vocabulary. However, the language models recline to cover major
vocabularies via the word embedding parameters, in particular, for multilingual
language models that generally cover a significant part of their overall
learning parameters. In this work, we present a new compact embedding structure
to reduce the memory footprint of the pre-trained language models with a
sacrifice of up to 4% absolute accuracy. The embeddings vectors reconstruction
follows a set of subspace embeddings and an assignment procedure via the
contextual relationship among tokens from pre-trained language models. The
subspace embedding structure calibrates to masked language models, to evaluate
our compact embedding structure on similarity and textual entailment tasks,
sentence and paraphrase tasks. Our experimental evaluation shows that the
subspace embeddings achieve compression rates beyond 99.8% in comparison with
the original embeddings for the language models on XNLI and GLUE benchmark
suites.

我们提出了一种新的紧凑嵌入结构，通过对预训练语言模型中的标记之间的上下文关系进行一组子空间嵌入和分配过程，来减少预训练语言模型的内存占用，牺牲高达 4% 的绝对准确率。我们的实验结果表明，子空间嵌入在 XNLI 和 GLUE 基准套件上与原始嵌入相比，达到了超过 99.8% 的压缩率。