We investigate the expressivity and learning dynamics of bias-free ReLU
networks. We firstly show that two-layer bias-free ReLU networks have limited
expressivity: the only odd function two-layer bias-free ReLU networks can
express is a linear one. We then show that, under symmetry conditions on the
data, these networks have the same learning dynamics as linear networks. This
allows us to give closed-form time-course solutions to certain two-layer
bias-free ReLU networks, which has not been done for nonlinear networks outside
the lazy learning regime. While deep bias-free ReLU networks are more
expressive than their two-layer counterparts, they still share a number of
similarities with deep linear networks. These similarities enable us to
leverage insights from linear networks, leading to a novel understanding of
bias-free ReLU networks. Overall, our results show that some properties
established for bias-free ReLU networks arise due to equivalence to linear
networks, and suggest that including bias or considering asymmetric data are
avenues to engage with nonlinear behaviors.

我们研究了无偏差 ReLU 网络的表现力和学习动态。我们首先展示了两层无偏差 ReLU 网络的有限表现力：它们只能表达线性的奇函数。然后我们证明，在数据对称条件下，这些网络具有与线性网络相同的学习动态，从而得到了某些特定两层无偏差 ReLU 网络的闭式时间课程解，这在非线性网络中尚未实现。虽然深层无偏差 ReLU 网络比两层网络具有更强的表现力，但它们仍然与深层线性网络有一些相似之处。这些相似之处使我们能够借鉴线性网络的见解，进而对无偏差 ReLU 网络有新的理解。总的来说，我们的结果表明，对于无偏差 ReLU 网络的某些属性是由于与线性网络等价而产生的，并且暗示加入偏差或考虑非对称数据是探索非线性行为的途径。