Conventional embedding methods directly associate each symbol with a continuous embedding vector, which is equivalent to applying a linear transformation based on a "one-hot" encoding of the discrete symbols. Despite its simplicity, such approach yields the number of parameters that grows linearly with the vocabulary size and can lead to overfitting. In this work, we propose a much more compact K-way D-dimensional discrete encoding scheme to replace the "one-hot" encoding. In the proposed "KD encoding", each symbol is represented by a $D$-dimensional code with a cardinality of $K$, and the final symbol embedding vector is generated by composing the code embedding vectors. To end-to-end learn semantically meaningful codes, we derive a relaxed discrete optimization approach based on stochastic gradient descent, which can be generally applied to any differentiable computational graph with an embedding layer. In our experiments with various applications from natural language processing to graph convolutional networks, the total size of the embedding layer can be reduced up to 98\% while achieving similar or better performance.

文中提出一种替换“one-hot”编码的更紧凑的K路D维离散编码方案，称为“KD编码”。通过基于随机梯度下降的松弛离散优化方法，可以端到端地学习语义上有意义的编码，实验证明该方案在各种自然语言处理和图卷积网络应用中，可以将嵌入层的总大小减少高达98％，同时实现类似或更好的性能。

学习 K-way D 维离散码以实现紧凑的嵌入表示