In this paper, we advance the understanding of neural network training
dynamics by examining the intricate interplay of various factors introduced by
weight parameters in the initialization process. Motivated by the foundational
work of Luo et al. (J. Mach. Learn. Res., Vol. 22, Iss. 1, No. 71, pp
3327-3373), we explore the gradient descent dynamics of neural networks through
the lens of macroscopic limits, where we analyze its behavior as width $m$
tends to infinity. Our study presents a unified approach with refined
techniques designed for multi-layer fully connected neural networks, which can
be readily extended to other neural network architectures. Our investigation
reveals that gradient descent can rapidly drive deep neural networks to zero
training loss, irrespective of the specific initialization schemes employed by
weight parameters, provided that the initial scale of the output function
$\kappa$ surpasses a certain threshold. This regime, characterized as the
theta-lazy area, accentuates the predominant influence of the initial scale
$\kappa$ over other factors on the training behavior of neural networks.
Furthermore, our approach draws inspiration from the Neural Tangent Kernel
(NTK) paradigm, and we expand its applicability. While NTK typically assumes
that $\lim_{m\to\infty}\frac{\log \kappa}{\log m}=\frac{1}{2}$, and imposes
each weight parameters to scale by the factor $\frac{1}{\sqrt{m}}$, in our
theta-lazy regime, we discard the factor and relax the conditions to
$\lim_{m\to\infty}\frac{\log \kappa}{\log m}>0$. Similar to NTK, the behavior
of overparameterized neural networks within the theta-lazy regime trained by
gradient descent can be effectively described by a specific kernel. Through
rigorous analysis, our investigation illuminates the pivotal role of $\kappa$
in governing the training dynamics of neural networks.

通过研究神经网络训练动态，探究权重参数在初始化过程中引入的各种因素之间的复杂相互作用，我们发现梯度下降可以快速将深层神经网络驱动至零训练误差，不论具体的初始化方案如何，只要输出函数的初始尺度超过一定阈值。这个区域被称为 theta-lazy 区域，突出了初始尺度在神经网络训练行为中的主导影响，同时我们还发现这个 theta-lazy 区域与神经切线核（NTK）范式有相似之处，但条件宽松。通过严格的分析，我们揭示了初始尺度 kappa 在神经网络训练动态中的关键作用。

从宏观角度解密神经网络的懒惰训练

Demystifying Lazy Training of Neural Networks from a Macroscopic  Viewpoint

Power consumption plays an important role in on-device streaming speech
recognition, as it has a direct impact on the user experience. This study
delves into how weight parameters in speech recognition models influence the
overall power consumption of these models. We discovered that the impact of
weight parameters on power consumption varies, influenced by factors including
how often they are invoked and their placement in memory. Armed with this
insight, we developed design guidelines aimed at optimizing on-device speech
recognition models. These guidelines focus on minimizing power use without
substantially affecting accuracy. Our method, which employs targeted
compression based on the varying sensitivities of weight parameters,
demonstrates superior performance compared to state-of-the-art compression
methods. It achieves a reduction in energy usage of up to 47% while maintaining
similar model accuracy and improving the real-time factor.

在本研究中，我们探讨了权重参数对语音识别模型整体功率消耗的影响，发现权重参数对功率消耗的影响因多种因素而异，包括调用频率和内存中的位置。基于这一洞察，我们制定了旨在优化设备内语音识别模型的设计准则，着重于在不实质影响准确性的情况下最小化功耗。我们的方法采用针对不同权重参数敏感性的有针对性压缩，相比于最先进的压缩方法，它在保持类似模型准确性的同时，实现了最高 47% 的能量使用减少，并改善了实时性。

不是所有的权重都是平等的：增强设备上流式语音识别的能效

Not All Weights Are Created Equal: Enhancing Energy Efficiency in  On-Device Streaming Speech Recognition

Not all neural network architectures are created equal, some perform much
better than others for certain tasks. But how important are the weight
parameters of a neural network compared to its architecture? In this work, we
question to what extent neural network architectures alone, without learning
any weight parameters, can encode solutions for a given task. We propose a
search method for neural network architectures that can already perform a task
without any explicit weight training. To evaluate these networks, we populate
the connections with a single shared weight parameter sampled from a uniform
random distribution, and measure the expected performance. We demonstrate that
our method can find minimal neural network architectures that can perform
several reinforcement learning tasks without weight training. On a supervised
learning domain, we find network architectures that achieve much higher than
chance accuracy on MNIST using random weights. Interactive version of this
paper at this https URL

本研究提出了一种搜索方法，寻找可以无需显式权重训练即可执行任务的神经网络框架，并证明该方法可以找到最小的神经网络架构，无需权重训练即可执行多个强化学习任务。