Iterative differential approximation methods that rely upon backpropagation
have enabled the optimization of neural networks; however, at present, they
remain computationally expensive, especially when training models at scale. In
this paper, we propose a computationally efficient alternative for optimizing
neural networks that can both reduce the costs of scaling neural networks and
provide high-efficiency optimizations for low-resource applications. We derive
an explicit solution to a simple feed-forward language model (LM) by
mathematically analyzing its gradients. This solution generalizes from
single-layer LMs to the class of all single-layer feed-forward
softmax-activated neural models trained on positive-valued features, as is
demonstrated by our extension of this solution application to MNIST digit
classification. For both LM and digit classifiers, we find computationally that
explicit solutions perform near-optimality in experiments showing that 1)
iterative optimization only marginally improves the explicit solution
parameters and 2) randomly initialized parameters iteratively optimize towards
the explicit solution. We also preliminarily apply the explicit solution
locally by layer in multi-layer networks and discuss how the solution's
computational savings increase with model complexity -- for both single- and
mult-layer applications of the explicit solution, we emphasize that the optima
achieved cannot be reached by backpropagation alone, i.e., better optima appear
discoverable only after explicit solutions are applied. Finally, we discuss the
solution's computational savings alongside its impact on model interpretability
and suggest future directions for the derivation of explicit solutions to
complex- and multi-layer architectures.

基于反向传播的迭代微分逼近方法使得神经网络的优化成为可能，但目前仍然计算代价高昂，尤其是在大规模训练模型时。本文提出了一种计算效率高的神经网络优化替代方案，既能降低神经网络的扩展成本，又能为低资源应用提供高效的优化。通过数学分析其梯度，我们推导了一个明确的解决方案用于简单的前馈语言模型 (LM)。该解决方案可以推广到基于正值特征训练的所有单层前馈 softmax 激活神经模型，我们通过将该解决方案应用于 MNIST 数字分类问题进行了验证。在 LM 和数字分类器的实验中，我们发现在计算上，明确的解决方案可以接近最优解，同时证明了：1) 迭代优化对明确解决方案参数的改进仅有微小影响，2) 随机初始化参数通过迭代优化逐渐趋向于明确的解决方案。我们还初步将明确的解决方案局部应用于多层网络，并讨论了随着模型复杂性增加的解决方案的计算节约。对于明确解决方案的单层和多层应用，我们强调仅通过反向传播无法达到这些最优解，即只有在应用明确解决方案后才能发现更好的最优解。最后，我们讨论了解决方案的计算节约以及它对模型可解释性的影响，并提出了为推导复杂和多层体系结构的明确解决方案的未来方向。