This paper pertains to an emerging machine learning paradigm: learning
higher-order functions, i.e. functions whose inputs are functions themselves,
$\textit{particularly when these inputs are Neural Networks (NNs)}$. With the
growing interest in architectures that process NNs, a recurring design
principle has permeated the field: adhering to the permutation symmetries
arising from the connectionist structure of NNs. $\textit{However, are these
the sole symmetries present in NN parameterizations}$? Zooming into most
practical activation functions (e.g. sine, ReLU, tanh) answers this question
negatively and gives rise to intriguing new symmetries, which we collectively
refer to as $\textit{scaling symmetries}$, that is, non-zero scalar
multiplications and divisions of weights and biases. In this work, we propose
$\textit{Scale Equivariant Graph MetaNetworks - ScaleGMNs}$, a framework that
adapts the Graph Metanetwork (message-passing) paradigm by incorporating
scaling symmetries and thus rendering neuron and edge representations
equivariant to valid scalings. We introduce novel building blocks, of
independent technical interest, that allow for equivariance or invariance with
respect to individual scalar multipliers or their product and use them in all
components of ScaleGMN. Furthermore, we prove that, under certain expressivity
conditions, ScaleGMN can simulate the forward and backward pass of any input
feedforward neural network. Experimental results demonstrate that our method
advances the state-of-the-art performance for several datasets and activation
functions, highlighting the power of scaling symmetries as an inductive bias
for NN processing.

该论文介绍了一种新兴的机器学习范式：学习高阶函数，特别是当这些函数的输入是神经网络时，尤其关注于与神经网络参数化中的排列对称性不同的扩展对称性，即缩放对称性。研究者提出了一种名为 ScaleGMNs 的框架，通过整合缩放对称性来使神经元和边的表示对于有效的缩放是等变的，并证明了它可以模拟任何输入前馈神经网络的前向和反向传递过程。实验结果表明，该方法在几个数据集和激活函数的性能上取得了领先的状态，突显了缩放对称性作为神经网络处理的一个归纳偏好的能力。