Machine learning tasks are generally formulated as optimization problems, where one searches for an optimal function within a certain functional space. In practice, parameterized functional spaces are considered, in order to be able to perform gradient descent. Typically, a neural network architecture is chosen and fixed, and its parameters (connection weights) are optimized, yielding an architecture-dependent result. This way of proceeding however forces the evolution of the function during training to lie within the realm of what is expressible with the chosen architecture, and prevents any optimization across architectures. Costly architectural hyper-parameter optimization is often performed to compensate for this. Instead, we propose to adapt the architecture on the fly during training. We show that the information about desirable architectural changes, due to expressivity bottlenecks when attempting to follow the functional gradient, can be extracted from %the backpropagation. To do this, we propose a mathematical definition of expressivity bottlenecks, which enables us to detect, quantify and solve them while training, by adding suitable neurons when and where needed. Thus, while the standard approach requires large networks, in terms of number of neurons per layer, for expressivity and optimization reasons, we are able to start with very small neural networks and let them grow appropriately. As a proof of concept, we show results~on the CIFAR dataset, matching large neural network accuracy, with competitive training time, while removing the need for standard architectural hyper-parameter search.

在这篇研究论文中，我们提出了一种新的机器学习方法，通过在训练过程中动态地调整神经网络的架构来解决架构选择和优化问题，从而避免了需要进行昂贵的架构参数优化。我们通过从反向传播中提取信息来检测和解决功能梯度路径上的表达能力瓶颈，从而实现可扩展性并优化训练过程，展示了在 CIFAR 数据集上与大型神经网络准确性相匹配的结果，并在竞争性训练时间内消除了标准架构超参数搜索的需要。

增长微小网络：发现并优化表达能力瓶颈