Cryo-EM is a vital technique for determining 3D structure of biological
molecules such as proteins and viruses. The cryo-EM reconstruction problem is
challenging due to the high noise levels, the missing poses of particles, and
the computational demands of processing large datasets. A promising solution to
these challenges lies in the use of amortized inference methods, which have
shown particular efficacy in pose estimation for large datasets. However, these
methods also encounter convergence issues, often necessitating sophisticated
initialization strategies or engineered solutions for effective convergence.
Building upon the existing cryoAI pipeline, which employs a symmetric loss
function to address convergence problems, this work explores the emergence and
persistence of these issues within the pipeline. Additionally, we explore the
impact of equivariant amortized inference on enhancing convergence. Our
investigations reveal that, when applied to simulated data, a pipeline
incorporating an equivariant encoder not only converges faster and more
frequently than the standard approach but also demonstrates superior
performance in terms of pose estimation accuracy and the resolution of the
reconstructed volume. Notably, $D_4$-equivariant encoders make the symmetric
loss superfluous and, therefore, allow for a more efficient reconstruction
pipeline.

冷冻电子显微术是一种确定蛋白质和病毒等生物分子三维结构的重要技术。本文探索了 cryoAI 管道中收敛问题的出现和持久性以及等变的摊解推理对增强收敛的影响，研究发现在模拟数据上，集成了等变编码器的管道不仅收敛更快、更频繁，而且在姿态估计准确度和重建体积的分辨率方面表现出优越性。值得注意的是，$D_4$- 等变编码器使对称损失变得多余，从而允许更高效的重建管道。

Cryo-EM 图像中姿态的等变化摊销推理

Equivariant amortized inference of poses for cryo-EM

Federated Averaging (FedAvg) is known to experience convergence issues when
encountering significant clients system heterogeneity and data heterogeneity.
Server momentum has been proposed as an effective mitigation. However, existing
server momentum works are restrictive in the momentum formulation, do not
properly schedule hyperparameters and focus only on system homogeneous
settings, which leaves the role of server momentum still an under-explored
problem. In this paper, we propose a general framework for server momentum,
that (a) covers a large class of momentum schemes that are unexplored in
federated learning (FL), (b) enables a popular stagewise hyperparameter
scheduler, (c) allows heterogeneous and asynchronous local computing. We
provide rigorous convergence analysis for the proposed framework. To our best
knowledge, this is the first work that thoroughly analyzes the performances of
server momentum with a hyperparameter scheduler and system heterogeneity.
Extensive experiments validate the effectiveness of our proposed framework.

提出了一种服务器动量的通用框架，用来解决联邦学习中由于客户端系统和数据异质性引起的收敛问题，并通过严密的收敛分析和大量实验证实了该框架的有效性。

关于服务器动量在联邦学习中的作用

On the Role of Server Momentum in Federated Learning

Existing approaches to federated learning suffer from a communication
bottleneck as well as convergence issues due to sparse client participation. In
this paper we introduce a novel algorithm, called FetchSGD, to overcome these
challenges. FetchSGD compresses model updates using a Count Sketch, and then
takes advantage of the mergeability of sketches to combine model updates from
many workers. A key insight in the design of FetchSGD is that, because the
Count Sketch is linear, momentum and error accumulation can both be carried out
within the sketch. This allows the algorithm to move momentum and error
accumulation from clients to the central aggregator, overcoming the challenges
of sparse client participation while still achieving high compression rates and
good convergence. We prove that FetchSGD has favorable convergence guarantees,
and we demonstrate its empirical effectiveness by training two residual
networks and a transformer model.

本文介绍了一种名为 FetchSGD 的算法，该算法通过使用 Count Sketch 压缩模型更新，并利用 Sketch 合并性从众多工作者中组合模型更新来克服联合学习中的通信瓶颈和因稀疏客户参与而导致的收敛问题。我们证明了 FetchSGD 具有良好的收敛性能，并通过训练两个残差网络和一个 Transformer 模型来证明其实证效果。

FetchSGD: 基于草图技术的高效通信联邦学习算法

FetchSGD: Communication-Efficient Federated Learning with Sketching

Several recently proposed stochastic optimization methods that have been
successfully used in training deep networks such as RMSProp, Adam, Adadelta,
Nadam are based on using gradient updates scaled by square roots of exponential
moving averages of squared past gradients. In many applications, e.g. learning
with large output spaces, it has been empirically observed that these
algorithms fail to converge to an optimal solution (or a critical point in
nonconvex settings). We show that one cause for such failures is the
exponential moving average used in the algorithms. We provide an explicit
example of a simple convex optimization setting where Adam does not converge to
the optimal solution, and describe the precise problems with the previous
analysis of Adam algorithm. Our analysis suggests that the convergence issues
can be fixed by endowing such algorithms with `long-term memory' of past
gradients, and propose new variants of the Adam algorithm which not only fix
the convergence issues but often also lead to improved empirical performance.

通过给 Adam 算法加上‘长期记忆’过去梯度的方法，不仅可以解决收敛问题，而且经常提高算法的实验性能。