This paper presents two simple improvements to the Self-Structuring
AutoEncoder (Self-StrAE). Firstly, we show that including reconstruction to the
vocabulary as an auxiliary objective improves representation quality. Secondly,
we demonstrate that increasing the number of independent channels leads to
significant improvements in embedding quality, while simultaneously reducing
the number of parameters. Surprisingly, we demonstrate that this trend can be
followed to the extreme, even to point of reducing the total number of
non-embedding parameters to seven. Our system can be pre-trained from scratch
with as little as 10M tokens of input data, and proves effective across
English, Spanish and Afrikaans.

该论文提出了两个简单的改进措施，一是将重构作为辅助目标包含在词汇中以提高表示质量，二是演示了增加独立通道数量可显著提高嵌入质量，同时减少参数数量，甚至将非嵌入参数总数减少到七的极限。我们的系统可以从头开始使用仅为 1000 万个输入数据进行预训练，并且在英语、西班牙语和南非荷兰语之间都表现出良好的效果。