Deep neural networks come in many sizes and architectures. The choice of
architecture, in conjunction with the dataset and learning algorithm, is
commonly understood to affect the learned neural representations. Yet, recent
results have shown that different architectures learn representations with
striking qualitative similarities. Here we derive an effective theory of
representation learning under the assumption that the encoding map from input
to hidden representation and the decoding map from representation to output are
arbitrary smooth functions. This theory schematizes representation learning
dynamics in the regime of complex, large architectures, where hidden
representations are not strongly constrained by the parametrization. We show
through experiments that the effective theory describes aspects of
representation learning dynamics across a range of deep networks with different
activation functions and architectures, and exhibits phenomena similar to the
"rich" and "lazy" regime. While many network behaviors depend quantitatively on
architecture, our findings point to certain behaviors that are widely conserved
once models are sufficiently flexible.

在复杂的大规模架构中，深度神经网络的表征学习动态可以用编解码映射为任意平滑函数的有效理论来描述，该理论能够概括多种不同激活函数和架构的深度网络的表征学习动态，并展现类似于 “丰富” 和 “懒惰” 区域的现象。

当表示对齐时：在表示学习动态中的普适性

When Representations Align: Universality in Representation Learning  Dynamics

We provide a construction for categorical representation learning and
introduce the foundations of "$\textit{categorifier}$". The central theme in
representation learning is the idea of $\textbf{everything to vector}$. Every
object in a dataset $\mathcal{S}$ can be represented as a vector in
$\mathbb{R}^n$ by an $\textit{encoding map}$ $E:
\mathcal{O}bj(\mathcal{S})\to\mathbb{R}^n$. More importantly, every morphism
can be represented as a matrix $E:
\mathcal{H}om(\mathcal{S})\to\mathbb{R}^{n}_{n}$. The encoding map $E$ is
generally modeled by a $\textit{deep neural network}$. The goal of
representation learning is to design appropriate tasks on the dataset to train
the encoding map (assuming that an encoding is optimal if it universally
optimizes the performance on various tasks). However, the latter is still a
$\textit{set-theoretic}$ approach. The goal of the current article is to
promote the representation learning to a new level via a
$\textit{category-theoretic}$ approach. As a proof of concept, we provide an
example of a text translator equipped with our technology, showing that our
categorical learning model outperforms the current deep learning models by 17
times. The content of the current article is part of the recent US patent
proposal (patent application number: 63110906).

本文介绍了一种基于范畴论的新的表示学习方法，并给出了一个例子：一种使用该方法的文本翻译器在性能上比当前深度学习模型高出了 17 倍。