This paper pertains to an emerging machine learning paradigm: learning
higher-order functions, i.e. functions whose inputs are functions themselves,
$\textit{particularly when these inputs are Neural Networks (NNs)}$. With the
growing interest in architectures that process NNs, a recurring design
principle has permeated the field: adhering to the permutation symmetries
arising from the connectionist structure of NNs. $\textit{However, are these
the sole symmetries present in NN parameterizations}$? Zooming into most
practical activation functions (e.g. sine, ReLU, tanh) answers this question
negatively and gives rise to intriguing new symmetries, which we collectively
refer to as $\textit{scaling symmetries}$, that is, non-zero scalar
multiplications and divisions of weights and biases. In this work, we propose
$\textit{Scale Equivariant Graph MetaNetworks - ScaleGMNs}$, a framework that
adapts the Graph Metanetwork (message-passing) paradigm by incorporating
scaling symmetries and thus rendering neuron and edge representations
equivariant to valid scalings. We introduce novel building blocks, of
independent technical interest, that allow for equivariance or invariance with
respect to individual scalar multipliers or their product and use them in all
components of ScaleGMN. Furthermore, we prove that, under certain expressivity
conditions, ScaleGMN can simulate the forward and backward pass of any input
feedforward neural network. Experimental results demonstrate that our method
advances the state-of-the-art performance for several datasets and activation
functions, highlighting the power of scaling symmetries as an inductive bias
for NN processing.

该论文介绍了一种新兴的机器学习范式：学习高阶函数，特别是当这些函数的输入是神经网络时，尤其关注于与神经网络参数化中的排列对称性不同的扩展对称性，即缩放对称性。研究者提出了一种名为 ScaleGMNs 的框架，通过整合缩放对称性来使神经元和边的表示对于有效的缩放是等变的，并证明了它可以模拟任何输入前馈神经网络的前向和反向传递过程。实验结果表明，该方法在几个数据集和激活函数的性能上取得了领先的状态，突显了缩放对称性作为神经网络处理的一个归纳偏好的能力。

尺度等变图元网络

Scale Equivariant Graph Metanetworks

Neural networks efficiently encode learned information within their
parameters. Consequently, many tasks can be unified by treating neural networks
themselves as input data. When doing so, recent studies demonstrated the
importance of accounting for the symmetries and geometry of parameter spaces.
However, those works developed architectures tailored to specific networks such
as MLPs and CNNs without normalization layers, and generalizing such
architectures to other types of networks can be challenging. In this work, we
overcome these challenges by building new metanetworks - neural networks that
take weights from other neural networks as input. Put simply, we carefully
build graphs representing the input neural networks and process the graphs
using graph neural networks. Our approach, Graph Metanetworks (GMNs),
generalizes to neural architectures where competing methods struggle, such as
multi-head attention layers, normalization layers, convolutional layers, ResNet
blocks, and group-equivariant linear layers. We prove that GMNs are expressive
and equivariant to parameter permutation symmetries that leave the input neural
network functions unchanged. We validate the effectiveness of our method on
several metanetwork tasks over diverse neural network architectures.

通过构建图形元网络，将权重从其他神经网络作为输入，我们的 Graph Metanetworks (GMNs) 方法解决了处理对称性和参数空间几何的困难，有效地推广到多种神经网络架构，并验证了其有效性。