We show that training deep neural networks (DNNs) with absolute value activation and arbitrary input dimension can be formulated as equivalent convex Lasso problems with novel features expressed using geometric algebra. This formulation reveals geometric structures encoding symmetry in neural networks. Using the equivalent Lasso form of DNNs, we formally prove a fundamental distinction between deep and shallow networks: deep networks inherently favor symmetric structures in their fitted functions, with greater depth enabling multilevel symmetries, i.e., symmetries within symmetries. Moreover, Lasso features represent distances to hyperplanes that are reflected across training points. These reflection hyperplanes are spanned by training data and are orthogonal to optimal weight vectors. Numerical experiments support theory and demonstrate theoretically predicted features when training networks using embeddings generated by Large Language Models.

本研究解决了深度神经网络（DNN）的训练中存在的对称性问题，提出将其作为等效的凸Lasso问题。研究表明，深层网络在拟合函数中自然偏向对称结构，且更大的深度可以实现多层次的对称性。这一发现对于理解深度网络的几何结构及其优化具有重要意义。

黑箱与镜子：深度网络中的多层对称性、反射平面与凸优化