Learning representations of well-trained neural network models holds the
promise to provide an understanding of the inner workings of those models.
However, previous work has either faced limitations when processing larger
networks or was task-specific to either discriminative or generative tasks.
This paper introduces the SANE approach to weight-space learning. SANE
overcomes previous limitations by learning task-agnostic representations of
neural networks that are scalable to larger models of varying architectures and
that show capabilities beyond a single task. Our method extends the idea of
hyper-representations towards sequential processing of subsets of neural
network weights, thus allowing one to embed larger neural networks as a set of
tokens into the learned representation space. SANE reveals global model
information from layer-wise embeddings, and it can sequentially generate unseen
neural network models, which was unattainable with previous
hyper-representation learning methods. Extensive empirical evaluation
demonstrates that SANE matches or exceeds state-of-the-art performance on
several weight representation learning benchmarks, particularly in
initialization for new tasks and larger ResNet architectures.

SANE 是一种权重空间学习方法，通过对神经网络的子集进行顺序处理，将较大的神经网络嵌入到学习表示空间中，并从层次嵌入中揭示全局模型信息，可生成新的神经网络模型，并在权重表示学习基准测试中达到或超过现有最先进方法的性能。

朝可扩展和多用途的权重空间学习迈进

Towards Scalable and Versatile Weight Space Learning

Self-Supervised Learning (SSL) has been shown to learn useful and
information-preserving representations. Neural Networks (NNs) are widely
applied, yet their weight space is still not fully understood. Therefore, we
propose to use SSL to learn hyper-representations of the weights of populations
of NNs. To that end, we introduce domain specific data augmentations and an
adapted attention architecture. Our empirical evaluation demonstrates that
self-supervised representation learning in this domain is able to recover
diverse NN model characteristics. Further, we show that the proposed learned
representations outperform prior work for predicting hyper-parameters, test
accuracy, and generalization gap as well as transfer to out-of-distribution
settings.

本文提出了在神经网络中使用自监督学习方法来学习超级表示法，引入了特定领域的数据增强和适应的注意力架构，并通过实验证明该方法能够较好地预测超参数、测试精度和泛化差距，并能用于超越分布场景。