Neural networks posses the crucial ability to generate meaningful
representations of task-dependent features. Indeed, with appropriate scaling,
supervised learning in neural networks can result in strong, task-dependent
feature learning. However, the nature of the emergent representations, which we
call the `coding scheme', is still unclear. To understand the emergent coding
scheme, we investigate fully-connected, wide neural networks learning
classification tasks using the Bayesian framework where learning shapes the
posterior distribution of the network weights. Consistent with previous
findings, our analysis of the feature learning regime (also known as
`non-lazy', `rich', or `mean-field' regime) shows that the networks acquire
strong, data-dependent features. Surprisingly, the nature of the internal
representations depends crucially on the neuronal nonlinearity. In linear
networks, an analog coding scheme of the task emerges. Despite the strong
representations, the mean predictor is identical to the lazy case. In nonlinear
networks, spontaneous symmetry breaking leads to either redundant or sparse
coding schemes. Our findings highlight how network properties such as scaling
of weights and neuronal nonlinearity can profoundly influence the emergent
representations.

神经网络具有生成任务相关特征有意义表示的关键能力，然而 emergent coding scheme 的本质仍不清楚。本文使用贝叶斯框架研究全连接的宽神经网络在学习分类任务时的特征学习机制，并发现网络的内部表示在神经非线性方面具有重要影响，线性网络呈现任务的模拟编码方案，而非线性网络则呈现冗余或稀疏编码方案。