Equivariant neural networks have shown improved performance, expressiveness
and sample complexity on symmetrical domains. But for some specific symmetries,
representations, and choice of coordinates, the most common point-wise
activations, such as ReLU, are not equivariant, hence they cannot be employed
in the design of equivariant neural networks. The theorem we present in this
paper describes all possible combinations of finite-dimensional
representations, choice of coordinates and point-wise activations to obtain an
exactly equivariant layer, generalizing and strengthening existing
characterizations. Notable cases of practical relevance are discussed as
corollaries. Indeed, we prove that rotation-equivariant networks can only be
invariant, as it happens for any network which is equivariant with respect to
connected compact groups. Then, we discuss implications of our findings when
applied to important instances of exactly equivariant networks. First, we
completely characterize permutation equivariant networks such as Invariant
Graph Networks with point-wise nonlinearities and their geometric counterparts,
highlighting a plethora of models whose expressive power and performance are
still unknown. Second, we show that feature spaces of disentangled steerable
convolutional neural networks are trivial representations.

等变神经网络在对称域上显示出了改进的性能、表现力和样本复杂度，但对于某些特定的对称性、表示和坐标选择，常见的逐点激活（如 ReLU）不能实现等变性，则无法用于等变神经网络的设计。本文中我们提出的定理描述了所有可能的有限维表示、坐标选择和逐点激活的组合，以获得完全等变的层，这样可以广义地推广和加强现有的特征。值得注意的是，实际相关的特殊情况被作为必要的推论进行了讨论。事实上，我们证明旋转等变网络只能是不变的，因为对于任何与连通紧致群等变的网络来说都是如此。然后，我们讨论了当应用于恰好等变网络的重要实例时，我们的研究结果的影响。首先，我们对诸如具有逐点非线性和几何对应物的不变图网络之类的置换等变网络进行了完全表征，突出了一大批具有尚不清楚的表现能力和性能的模型。其次，我们表明了解离可操作卷积神经网络的特征空间是平凡表示。