Huge embedding tables in modern Deep Learning Recommender Models (DLRM)
require prohibitively large memory during training and inference. Aiming to
reduce the memory footprint of training, this paper proposes FIne-grained
In-Training Embedding Dimension optimization (FIITED). Given the observation
that embedding vectors are not equally important, FIITED adjusts the dimension
of each individual embedding vector continuously during training, assigning
longer dimensions to more important embeddings while adapting to dynamic
changes in data. A novel embedding storage system based on virtually-hashed
physically-indexed hash tables is designed to efficiently implement the
embedding dimension adjustment and effectively enable memory saving.
Experiments on two industry models show that FIITED is able to reduce the size
of embeddings by more than 65% while maintaining the trained model's quality,
saving significantly more memory than a state-of-the-art in-training embedding
pruning method. On public click-through rate prediction datasets, FIITED is
able to prune up to 93.75%-99.75% embeddings without significant accuracy loss.

通过在训练期间动态调整嵌入向量的维度，使用虚拟散列的物理索引哈希表设计了一种新颖的嵌入存储系统，从而实现内存节省。实验结果表明，该方法在减少嵌入大小超过 65% 的同时，保持了训练模型的质量，并且相比最先进的嵌入剪枝方法，能够节省更多的内存。在公共点击率预测数据集上，能够剪枝 93.75%-99.75% 的嵌入而几乎不影响准确性。