Implicit neural representations have emerged as a powerful technique for
encoding complex continuous multidimensional signals as neural networks,
enabling a wide range of applications in computer vision, robotics, and
geometry. While Adam is commonly used for training due to its stochastic
proficiency, it entails lengthy training durations. To address this, we explore
alternative optimization techniques for accelerated training without
sacrificing accuracy. Traditional second-order optimizers like L-BFGS are
suboptimal in stochastic settings, making them unsuitable for large-scale data
sets. Instead, we propose stochastic training using curvature-aware diagonal
preconditioners, showcasing their effectiveness across various signal
modalities such as images, shape reconstruction, and Neural Radiance Fields
(NeRF).

利用曲率感知的对角预处理器，我们提出了一种用于加速训练的随机训练方法，展示了它们在图像、形状重建和神经辐射场等各种信号模态上的有效性。

隐式神经表示的随机训练预处理器

Preconditioners for the Stochastic Training of Implicit Neural  Representations

In this paper, we study zeroth-order algorithms for nonconvex minimax
problems with coupled linear constraints under the deterministic and stochastic
settings, which have attracted wide attention in machine learning, signal
processing and many other fields in recent years, e.g., adversarial attacks in
resource allocation problems and network flow problems etc. We propose two
single-loop algorithms, namely the zero-order primal-dual alternating projected
gradient (ZO-PDAPG) algorithm and the zero-order regularized momentum
primal-dual projected gradient algorithm (ZO-RMPDPG), for solving deterministic
and stochastic nonconvex-(strongly) concave minimax problems with coupled
linear constraints. The iteration complexity of the two proposed algorithms to
obtain an $\varepsilon$-stationary point are proved to be
$\mathcal{O}(\varepsilon ^{-2})$ (resp. $\mathcal{O}(\varepsilon ^{-4})$) for
solving nonconvex-strongly concave (resp. nonconvex-concave) minimax problems
with coupled linear constraints under deterministic settings and
$\tilde{\mathcal{O}}(\varepsilon ^{-3})$ (resp.
$\tilde{\mathcal{O}}(\varepsilon ^{-6.5})$) under stochastic settings
respectively. To the best of our knowledge, they are the first two zeroth-order
algorithms with iterative complexity guarantees for solving
nonconvex-(strongly) concave minimax problems with coupled linear constraints
under the deterministic and stochastic settings.

研究零阶算法求解非凸最小最大问题在确定性和随机设置下的紧密线性约束，为确定性和随机设置下求解非凸凹最小最大问题提出两种单环算法，即零阶原始 -- 对偶交替投影梯度（ZO-PDAPG）算法和零阶正则化动量原始 -- 对偶投影梯度算法（ZO-RMPDPG），其迭代复杂度被证明为 Ο(ε^(-2))（非凸凹最小最大问题）和 Ο(ε^(-4))（非凸凹最小最大问题），在确定性设置下，持有 ε- 稳定点，并且在随机设置下，迭代复杂度分别为 Ο̃(ε^(-3)) 和 Ο̃(ε^(-6.5))。据我们所知，它们是第一个能够解决确定性和随机设置下的非凸凹最小最大问题的零阶算法，并具有迭代复杂度保证。

零阶原始 - 对偶交替投影梯度算法用于带有耦合线性约束的非凸极小极大问题

Zeroth-Order primal-dual Alternating Projection Gradient Algorithms for  Nonconvex Minimax Problems with Coupled linear Constraints

The natural gradient method has been used effectively in conjugate Gaussian
process models, but the non-conjugate case has been largely unexplored. We
examine how natural gradients can be used in non-conjugate stochastic settings,
together with hyperparameter learning. We conclude that the natural gradient
can significantly improve performance in terms of wall-clock time. For
ill-conditioned posteriors the benefit of the natural gradient method is
especially pronounced, and we demonstrate a practical setting where ordinary
gradients are unusable. We show how natural gradients can be computed
efficiently and automatically in any parameterization, using automatic
differentiation. Our code is integrated into the GPflow package.

本文探讨了利用自然梯度方法在非共轭随机模型环境下的超参数学习，结果显示自然梯度方法可显著提高性能和效率并已被集成于 GPflow 软件包中。