Optimization algorithms is crucial in training physics-informed neural
networks (PINNs), unsuitable methods may lead to poor solutions. Compared to
the common gradient descent algorithm, implicit gradient descent (IGD)
outperforms it in handling some multi-scale problems. In this paper, we provide
convergence analysis for the implicit gradient descent for training
over-parametrized two-layer PINNs. We first demonstrate the positive
definiteness of Gram matrices for general smooth activation functions, like
sigmoidal function, softplus function, tanh function and so on. Then the
over-parameterization allows us to show that the randomly initialized IGD
converges a globally optimal solution at a linear convergence rate. Moreover,
due to the different training dynamics, the learning rate of IGD can be chosen
independent of the sample size and the least eigenvalue of the Gram matrix.

本文提供了用于训练过参数化的两层物理信息神经网络的隐性梯度下降法的收敛性分析，证明了常见的平滑激活函数（如 sigmoid 函数、softplus 函数、tanh 函数等）的 Gram 矩阵是正定的。通过过参数化，随机初始化的隐性梯度下降法以线性收敛速率收敛于全局最优解，并且由于不同的训练动态，可以独立于样本大小和 Gram 矩阵的最小特征值选择学习率。

训练两层物理引导神经网络的隐式梯度下降收敛性

Convergence of Implicit Gradient Descent for Training Two-Layer  Physics-Informed Neural Networks

Pre-trained large language models (LLMs) based on Transformer have
demonstrated striking in-context learning (ICL) abilities. With a few
demonstration input-label pairs, they can predict the label for an unseen input
without any parameter updates. In this paper, we show an exciting phenomenon
that SVD-based weight pruning can enhance ICL performance, and more surprising,
pruning weights in deep layers often results in more stable performance
improvements in shallow layers. However, the underlying mechanism of those
findings still remains an open question. To reveal those findings, we conduct
an in-depth theoretical analysis by presenting the implicit gradient descent
(GD) trajectories of ICL and giving the mutual information based generalization
bounds of ICL via full implicit GD trajectories. This helps us reasonably
explain the surprising experimental findings. Besides, based on all our
experimental and theoretical insights, we intuitively propose a simple,
model-compression and derivative-free algorithm for downstream tasks in
enhancing ICL inference. Experiments on benchmark datasets and open source LLMs
display the method effectiveness\footnote{The code is available at
https://github.com/chen123CtrlS/EnhancingICL_SVDPruning}.

基于 Transfomer 的预训练大型语言模型（LLM）展示了令人惊叹的上下文学习能力（ICL）。在本文中，我们展示了基于 SVD 的权重剪枝可以增强 ICL 性能的有趣现象，并且在深层剪枝权重通常导致浅层性能的更稳定的改善。然而，这些发现的基本机制仍然是一个悬而未决的问题。为了揭示这些发现，我们通过展示 ICL 的隐式梯度下降（GD）轨迹，并通过完全的隐式 GD 轨迹给出基于互信息的 ICL 泛化界限进行了深入的理论分析。这有助于我们合理地解释令人惊讶的实验结果。此外，基于所有的实验和理论观察，我们直观地提出了一个用于增强 ICL 推断的简单、压缩模型和无导数的算法。在基准数据集和开源 LLM 上的实验证明了该方法的有效性。

基于 SVD 的加权剪枝提升上下文学习性能：理论视角

Enhancing In-Context Learning Performance with just SVD-Based Weight  Pruning: A Theoretical Perspective

In this paper, we propose an implicit gradient descent algorithm for the
classic $k$-means problem. The implicit gradient step or backward Euler is
solved via stochastic fixed-point iteration, in which we randomly sample a
mini-batch gradient in every iteration. It is the average of the fixed-point
trajectory that is carried over to the next gradient step. We draw connections
between the proposed stochastic backward Euler and the recent entropy
stochastic gradient descent (Entropy-SGD) for improving the training of deep
neural networks. Numerical experiments on various synthetic and real datasets
show that the proposed algorithm provides better clustering results compared to
$k$-means algorithms in the sense that it decreased the objective function (the
cluster) and is much more robust to initialization.

本文提出了一种用于经典 K 均值问题的隐式梯度下降算法：通过随机固定点迭代解决，当前迭代的平均固定点轨迹传递到下一梯度步骤，数值实验表明该算法相对于 K 均值算法在聚类结果和每次迭代的目标函数上有所提升，更为稳健。与最近的 Entropy-SGD 相比，具有类似性质。