We introduce EELBERT, an approach for compression of transformer-based models (e.g., BERT), with minimal impact on the accuracy of downstream tasks. This is achieved by replacing the input embedding layer of the model with dynamic, i.e. on-the-fly, embedding computations. Since the input embedding layer accounts for a significant fraction of the model size, especially for the smaller BERT variants, replacing this layer with an embedding computation function helps us reduce the model size significantly. Empirical evaluation on the GLUE benchmark shows that our BERT variants (EELBERT) suffer minimal regression compared to the traditional BERT models. Through this approach, we are able to develop our smallest model UNO-EELBERT, which achieves a GLUE score within 4% of fully trained BERT-tiny, while being 15x smaller (1.2 MB) in size.

EELBERT是一种基于transformer模型（例如BERT）的压缩方法，通过在模型的输入嵌入层中使用动态嵌入计算来实现对下游任务准确性的最小影响。通过替换嵌入层，我们成功地显著减小了模型的大小，并在GLUE基准测试中证明了EELBERT与传统BERT模型之间仅有微小的回归差异。这一方法使得我们能够开发出最小的模型UNO-EELBERT，其大小为14MB，GLUE得分仅相对于完全训练的BERT-tiny低4%。

EELBERT：通过动态嵌入实现微小模型