In this work, we present a naive initialization scheme for word vectors based
on a dense, independent co-occurrence model and provide preliminary results
that suggest it is competitive and warrants further investigation.
Specifically, we demonstrate through information-theoretic minimum description
length (MDL) probing that our model, EigenNoise, can approach the performance
of empirically trained GloVe despite the lack of any pre-training data (in the
case of EigenNoise). We present these preliminary results with interest to set
the stage for further investigations into how this competitive initialization
works without pre-training data, as well as to invite the exploration of more
intelligent initialization schemes informed by the theory of harmonic
linguistic structure. Our application of this theory likewise contributes a
novel (and effective) interpretation of recent discoveries which have
elucidated the underlying distributional information that linguistic
representations capture from data and contrast distributions.

本研究提出了一种基于稠密、独立共现模型的简单初始化方案，具有与经验训练的 GloVe 相当的性能，并探索了基于谐波语言结构理论的更智能初始化方案的可能性。