We explore conditions for when the gradient of a deep declarative node can be
approximated by ignoring constraint terms and still result in a descent
direction for the global loss function. This has important practical
application when training deep learning models since the approximation is often
computationally much more efficient than the true gradient calculation. We
provide theoretical analysis for problems with linear equality constraints and
normalization constraints, and show examples where the approximation works well
in practice as well as some cautionary tales for when it fails.

探讨当忽略约束项时，深层声明节点的梯度可以被近似，仍然产生全局损失函数的下降方向的条件。

深度声明网络中梯度近似性的理解

Towards Understanding Gradient Approximation in Equality Constrained  Deep Declarative Networks

Locally supervised learning aims to train a neural network based on a local
estimation of the global loss function at each decoupled module of the network.
Auxiliary networks are typically appended to the modules to approximate the
gradient updates based on the greedy local losses. Despite being advantageous
in terms of parallelism and reduced memory consumption, this paradigm of
training severely degrades the generalization performance of neural networks.
In this paper, we propose Periodically Guided local Learning (PGL), which
reinstates the global objective repetitively into the local-loss based training
of neural networks primarily to enhance the model's generalization capability.
We show that a simple periodic guidance scheme begets significant performance
gains while having a low memory footprint. We conduct extensive experiments on
various datasets and networks to demonstrate the effectiveness of PGL,
especially in the configuration with numerous decoupled modules.

提出一种周期性引导的本地化学习方法，旨在提升神经网络的泛化能力。该方法将全局目标函数反复引入神经网络的本地化损失训练中，并在多个模块的配置中具有重要的性能提升。

具有周期全局引导的本地监督学习

Locally Supervised Learning with Periodic Global Guidance

This paper addresses the problem of online learning in a dynamic setting. We
consider a social network in which each individual observes a private signal
about the underlying state of the world and communicates with her neighbors at
each time period. Unlike many existing approaches, the underlying state is
dynamic, and evolves according to a geometric random walk. We view the scenario
as an optimization problem where agents aim to learn the true state while
suffering the smallest possible loss. Based on the decomposition of the global
loss function, we introduce two update mechanisms, each of which generates an
estimate of the true state. We establish a tight bound on the rate of change of
the underlying state, under which individuals can track the parameter with a
bounded variance. Then, we characterize explicit expressions for the steady
state mean-square deviation(MSD) of the estimates from the truth, per
individual. We observe that only one of the estimators recovers the optimal
MSD, which underscores the impact of the objective function decomposition on
the learning quality. Finally, we provide an upper bound on the regret of the
proposed methods, measured as an average of errors in estimating the parameter
in a finite time.

研究在线学习在动态环境中的应用，通过考虑社交网络等输入变量来降低最小化损失函数，并建立针对真实状态的两个更新机制，并对估计误差和后悔进行建模。