Modern DNN-based recommendation systems rely on training-derived embeddings
of sparse features. Input sparsity makes obtaining high-quality embeddings for
rarely-occurring categories harder as their representations are updated
infrequently. We demonstrate a training-time technique to produce superior
embeddings via effective cross-category learning and theoretically explain its
surprising effectiveness. The scheme, termed the multi-layer embeddings
training (MLET), trains embeddings using factorization of the embedding layer,
with an inner dimension higher than the target embedding dimension. For
inference efficiency, MLET converts the trained two-layer embedding into a
single-layer one thus keeping inference-time model size unchanged.
Empirical superiority of MLET is puzzling as its search space is not larger
than that of the single-layer embedding. The strong dependence of MLET on the
inner dimension is even more surprising. We develop a theory that explains both
of these behaviors by showing that MLET creates an adaptive update mechanism
modulated by the singular vectors of embeddings. When tested on multiple
state-of-the-art recommendation models for click-through rate (CTR) prediction
tasks, MLET consistently produces better models, especially for rare items. At
constant model quality, MLET allows embedding dimension, and model size,
reduction by up to 16x, and 5.8x on average, across the models.

现代的基于 DNN 的推荐系统依赖于对稀疏特征进行训练得到的嵌入。输入的稀疏性使得对于很少出现的类别难以获得高质量的嵌入，因为它们的表示不经常更新。我们通过有效的跨类别学习在训练时演示了一种产生优越嵌入的技术，并从理论上解释了它的出人意料的有效性。这种称为多层嵌入训练（MLET）的方案使用嵌入层的分解来训练嵌入，内部维度高于目标嵌入维度。为了提高推理效率，MLET 将经过训练的双层嵌入转换为单层嵌入，从而保持推理时模型大小不变。在多个最先进的点击率（CTR）预测任务的推荐模型上进行测试时，MLET 始终产生更好的模型，特别是对于罕见的项目。在保持模型质量不变的情况下，MLET 允许模型的嵌入维度和模型大小分别减少高达 16 倍和平均减少 5.8 倍。