The fundamental success of large language models hinges upon the efficacious
implementation of large-scale distributed training techniques. Nevertheless,
building a vast, high-performance cluster featuring high-speed communication
interconnectivity is prohibitively costly, and accessible only to prominent
entities. In this work, we aim to lower this barrier and democratize
large-scale training with limited bandwidth clusters. We propose a new approach
called CO2 that introduces local-updating and asynchronous communication to the
distributed data-parallel training, thereby facilitating the full overlap of
COmunication with COmputation. CO2 is able to attain a high scalability even on
extensive multi-node clusters constrained by very limited communication
bandwidth. We further propose the staleness gap penalty and outer momentum
clipping techniques together with CO2 to bolster its convergence and training
stability. Besides, CO2 exhibits seamless integration with well-established
ZeRO-series optimizers which mitigate memory consumption of model states with
large model training. We also provide a mathematical proof of convergence,
accompanied by the establishment of a stringent upper bound. Furthermore, we
validate our findings through an extensive set of practical experiments
encompassing a wide range of tasks in the fields of computer vision and natural
language processing. These experiments serve to demonstrate the capabilities of
CO2 in terms of convergence, generalization, and scalability when deployed
across configurations comprising up to 128 A100 GPUs. The outcomes emphasize
the outstanding capacity of CO2 to hugely improve scalability, no matter on
clusters with 800Gbps RDMA or 80Gbps TCP/IP inter-node connections.

基于限制带宽的集群，我们提出了一种名为 CO2 的新方法，通过引入局部更新和异步通信实现分布式数据并行训练的完全重叠，从而实现大规模训练的高可扩展性。我们还提出了降低收敛性和训练稳定性的过时惩罚和外部动能裁剪技术。通过大量实验验证了 CO2 在计算机视觉和自然语言处理领域的各种任务上的收敛性、泛化性和可伸缩性，无论是在具有 800Gbps RDMA 还是 80Gbps TCP/IP 的集群中，CO2 都表现出卓越的能力来大幅提高可扩展性。

CO2：高效分布式训练与完全通信 - 计算重叠

CO2: Efficient Distributed Training with Full Communication-Computation  Overlap

Contrastive learning has been adopted as a core method for unsupervised
visual representation learning. Without human annotation, the common practice
is to perform an instance discrimination task: Given a query image crop, this
task labels crops from the same image as positives, and crops from other
randomly sampled images as negatives. An important limitation of this label
assignment strategy is that it can not reflect the heterogeneous similarity
between the query crop and each crop from other images, taking them as equally
negative, while some of them may even belong to the same semantic class as the
query. To address this issue, inspired by consistency regularization in
semi-supervised learning on unlabeled data, we propose Consistent Contrast
(CO2), which introduces a consistency regularization term into the current
contrastive learning framework. Regarding the similarity of the query crop to
each crop from other images as "unlabeled", the consistency term takes the
corresponding similarity of a positive crop as a pseudo label, and encourages
consistency between these two similarities. Empirically, CO2 improves Momentum
Contrast (MoCo) by 2.9% top-1 accuracy on ImageNet linear protocol, 3.8% and
1.1% top-5 accuracy on 1% and 10% labeled semi-supervised settings. It also
transfers to image classification, object detection, and semantic segmentation
on PASCAL VOC. This shows that CO2 learns better visual representations for
these downstream tasks.

该研究提出了一种新的无监督对比学习框架 CO2，它利用了模仿半监督学习中的一致性正则化来处理标签分配策略的瓶颈问题，能够提高图像分类、目标检测和语义分割等任务的视觉表示能力。