Recent advances in decentralized deep learning algorithms have demonstrated
cutting-edge performance on various tasks with large pre-trained models.
However, a pivotal prerequisite for achieving this level of competitiveness is
the significant communication and computation overheads when updating these
models, which prohibits the applications of them to real-world scenarios. To
address this issue, drawing inspiration from advanced model merging techniques
without requiring additional training, we introduce the Decentralized Iterative
Merging-And-Training (DIMAT) paradigm--a novel decentralized deep learning
framework. Within DIMAT, each agent is trained on their local data and
periodically merged with their neighboring agents using advanced model merging
techniques like activation matching until convergence is achieved. DIMAT
provably converges with the best available rate for nonconvex functions with
various first-order methods, while yielding tighter error bounds compared to
the popular existing approaches. We conduct a comprehensive empirical analysis
to validate DIMAT's superiority over baselines across diverse computer vision
tasks sourced from multiple datasets. Empirical results validate our
theoretical claims by showing that DIMAT attains faster and higher initial gain
in accuracy with independent and identically distributed (IID) and non-IID
data, incurring lower communication overhead. This DIMAT paradigm presents a
new opportunity for the future decentralized learning, enhancing its
adaptability to real-world with sparse and light-weight communication and
computation.

从先进的模型合并技术中汲取灵感，我们引入了分布式迭代融合和训练（DIMAT）范式 —— 一种新颖的分布式深度学习框架，它通过使用高级模型合并技术如激活匹配，使每个代理在本地数据上训练，并定期与相邻代理合并，直到达到收敛。实证分析验证了 DIMAT 在多个数据集上的计算机视觉任务中相对于基线方法的优越性，结果表明 DIMAT 在使用独立同分布（IID）和非 IID 数据时以更快的速度和更高的初始准确性获得收敛，并且通信开销更低。DIMAT 范式为未来的分布式学习提供了新的机会，增强了其适应真实世界的稀疏和轻量级通信和计算需求。