This article seeks for a distributed learning solution for the visual transformer (ViT) architectures. Compared to convolutional neural network (CNN) architectures, ViTs often have larger model sizes, and are computationally expensive, making federated learning (FL) ill-suited. Split learning (SL) can detour this problem by splitting a model and communicating the hidden representations at the split-layer, also known as smashed data. Notwithstanding, the smashed data of ViT are as large as and as similar as the input data, negating the communication efficiency of SL while violating data privacy. To resolve these issues, we propose a new form of CutSmashed data by randomly punching and compressing the original smashed data. Leveraging this, we develop a novel SL framework for ViT, coined CutMixSL, communicating CutSmashed data. CutMixSL not only reduces communication costs and privacy leakage, but also inherently involves the CutMix data augmentation, improving accuracy and scalability. Simulations corroborate that CutMixSL outperforms baselines such as parallelized SL and SplitFed that integrates FL with SL.

本文提出了一种名为CutMixSL的新型分布式学习框架，旨在解决视觉Transformer(简称ViT)架构中数据沟通效率低下和隐私泄露等问题。CutMixSL可以通过随机打孔和压缩原始数据来优化Smashed data的通讯，实现减少通讯成本，提高准确性和可伸缩性等目标。与现有的基线方案相比，CutMixSL的表现更优。

视觉Transformer遇上CutMix：在分割学习中提高准确性、通信效率和数据隐私