The separation power of a machine learning model refers to its capacity to distinguish distinct inputs, and it is often employed as a proxy for its expressivity. In this paper, we propose a theoretical framework to investigate the separation power of equivariant neural networks with point-wise activations. Using the proposed framework, we can derive an explicit description of inputs indistinguishable by a family of neural networks with given architecture, demonstrating that it remains unaffected by the choice of non-polynomial activation function employed. We are able to understand the role played by activation functions in separability. Indeed, we show that all non-polynomial activations, such as ReLU and sigmoid, are equivalent in terms of expressivity, and that they reach maximum discrimination capacity. We demonstrate how assessing the separation power of an equivariant neural network can be simplified to evaluating the separation power of minimal representations. We conclude by illustrating how these minimal components form a hierarchy in separation power.

我们提出了一个理论框架来研究具有点逐元激活的等变神经网络的分离能力，我们能够推导出一族神经网络对于给定架构的输入无法区分的显式描述，证明其不受所采用的非多项式激活函数选择的影响，我们证明了激活函数在可分性中的作用，所有非多项式激活函数，如ReLU和sigmoid，在表达能力方面是等价的，并且达到了最大的区分能力，我们演示了如何简化等变神经网络的分离能力评估为评估最小表示的分离能力，并且说明了这些最小组件如何形成分离能力的等级结构。