Learning continuous representations of discrete objects such as text, users, and URLs lies at the heart of many applications including language and user modeling. When using discrete objects as input to neural networks, we often ignore the underlying structures (e.g. natural groupings and similarities) and embed the objects independently into individual vectors. As a result, existing methods do not scale to large vocabulary sizes. In this paper, we design a Bayesian nonparametric prior for embeddings that encourages sparsity and leverages natural groupings among objects. We derive an approximate inference algorithm based on Small Variance Asymptotics which yields a simple and natural algorithm for learning a small set of anchor embeddings and a sparse transformation matrix. We call our method Anchor & Transform (ANT) as the embeddings of discrete objects are a sparse linear combination of the anchors, weighted according to the transformation matrix. ANT is scalable, flexible, end-to-end trainable, and allows the user to incorporate domain knowledge about object relationships. On text classification and language modeling benchmarks, ANT demonstrates stronger performance with fewer parameters as compared to existing compression baselines.

本文提出了一种名为“Anchor & Transform (ANT)”的简单而高效的嵌入算法，它能够处理大容量的离散对象嵌入，通过从中学习少量的锚点和稀疏的转换矩阵，将对象嵌入成为锚点的稀疏线性组合。这种算法是可扩展、灵活和端到端训练的。同时，研究人员还将其解释为一种贝叶斯非参数先验，旨在促进稀疏性并利用对象间的自然分组关系，我们在多个基准测试中发现，ANT特别适用于大词汇量，相较于现有压缩基线提供了更强的性能表现（高达40倍压缩的低参数）。

锚定&转换: 为大词汇量学习稀疏嵌入