We propose an approximate strategy to efficiently train neural network based
language models over very large vocabularies. Our approach, called adaptive
softmax, circumvents the linear dependency on the vocabulary size by exploiting
the unbalanced word distribution to form clusters that explicitly minimize the
expectation of computation time. Our approach further reduces the computational
time by exploiting the specificities of modern architectures and matrix-matrix
vector operations, making it particularly suited for graphical processing
units. Our experiments carried out on standard benchmarks, such as EuroParl and
One Billion Word, show that our approach brings a large gain in efficiency over
standard approximations while achieving an accuracy close to that of the full
softmax. The code of our method is available at
this https URL

文章提出了一种名为自适应 softmax 的算法，采用字词聚类的方法来降低神经网络语言模型训练中的计算复杂度，并结合现代计算机体系结构和矩阵向量运算技术进一步提高了训练效率。实验结果表明该方法的效果稳定，能够在保证高精度的同时显著提高计算速度。