This paper presents two simple improvements to the Self-Structuring
AutoEncoder (Self-StrAE). Firstly, we show that including reconstruction to the
vocabulary as an auxiliary objective improves representation quality. Secondly,
we demonstrate that increasing the number of independent channels leads to
significant improvements in embedding quality, while simultaneously reducing
the number of parameters. Surprisingly, we demonstrate that this trend can be
followed to the extreme, even to point of reducing the total number of
non-embedding parameters to seven. Our system can be pre-trained from scratch
with as little as 10M tokens of input data, and proves effective across
English, Spanish and Afrikaans.

该论文提出了两个简单的改进措施，一是将重构作为辅助目标包含在词汇中以提高表示质量，二是演示了增加独立通道数量可显著提高嵌入质量，同时减少参数数量，甚至将非嵌入参数总数减少到七的极限。我们的系统可以从头开始使用仅为 1000 万个输入数据进行预训练，并且在英语、西班牙语和南非荷兰语之间都表现出良好的效果。

Self-StrAE 在 SemEval-2024 任务 1 中：用更少地让自我结构化自编码器学习更多

Self-StrAE at SemEval-2024 Task 1: Making Self-Structuring AutoEncoders  Learn More With Less

Unsupervised learning has recently significantly gained in popularity,
especially with deep learning-based approaches. Despite numerous successes and
approaching supervised-level performance on a variety of academic benchmarks,
it is still hard to train and evaluate SSL models in practice due to the
unsupervised nature of the problem. Even with networks trained in a supervised
fashion, it is often unclear whether they will perform well when transferred to
another domain.
Past works are generally limited to assessing the amount of information
contained in embeddings, which is most relevant for self-supervised learning of
deep neural networks. This works chooses to follow a different approach: can we
quantify how easy it is to linearly separate the data in a stable way? We
survey the literature and uncover three methods that could be potentially used
for evaluating quality of representations. We also introduce one novel method
based on recent advances in understanding the high-dimensional geometric
structure self-supervised learning.
We conduct extensive experiments and study the properties of these metrics
and ones introduced in the previous work. Our results suggest that while there
is no free lunch, there are metrics that can robustly estimate embedding
quality in an unsupervised way.

研究了评估深度神经网络自监督学习中表示质量的多种方法，包括信息嵌入量、线性可分性等，通过实验结果发现了可以无监督评估嵌入质量的方法。

无监督嵌入质量评估

Unsupervised Embedding Quality Evaluation

Graph embedding techniques are a staple of modern graph learning research.
When using embeddings for downstream tasks such as classification, information
about their stability and robustness, i.e., their susceptibility to sources of
noise, stochastic effects, or specific parameter choices, becomes increasingly
important. As one of the most prominent graph embedding schemes, we focus on
node2vec and analyse its embedding quality from multiple perspectives. Our
findings indicate that embedding quality is unstable with respect to parameter
choices, and we propose strategies to remedy this in practice.

分析了 node2vec 图嵌入策略在多个方面的嵌入质量，发现其嵌入质量对参数选择不稳定，并提出了实际上的解决方案。