Federated learning (FL) has emerged as a promising approach to
collaboratively train machine learning models across multiple edge devices
while preserving privacy. The success of FL hinges on the efficiency of
participating models and their ability to handle the unique challenges of
distributed learning. While several variants of Vision Transformer (ViT) have
shown great potential as alternatives to modern convolutional neural networks
(CNNs) for centralized training, the unprecedented size and higher
computational demands hinder their deployment on resource-constrained edge
devices, challenging their widespread application in FL. Since client devices
in FL typically have limited computing resources and communication bandwidth,
models intended for such devices must strike a balance between model size,
computational efficiency, and the ability to adapt to the diverse and non-IID
data distributions encountered in FL. To address these challenges, we propose
OnDev-LCT: Lightweight Convolutional Transformers for On-Device vision tasks
with limited training data and resources. Our models incorporate image-specific
inductive biases through the LCT tokenizer by leveraging efficient depthwise
separable convolutions in residual linear bottleneck blocks to extract local
features, while the multi-head self-attention (MHSA) mechanism in the LCT
encoder implicitly facilitates capturing global representations of images.
Extensive experiments on benchmark image datasets indicate that our models
outperform existing lightweight vision models while having fewer parameters and
lower computational demands, making them suitable for FL scenarios with data
heterogeneity and communication bottlenecks.

提出了 OnDev-LCT：适用于具有有限训练数据和资源的设备上的轻量级卷积变换器，该模型通过利用残差线性瓶颈块中的高效深度可分离卷积来提取局部特征，并通过 LCT 编码器中的多头自注意机制隐式地捕获图像的全局表示。大量基准图像数据集上的实验证明，我们的模型在拥有更少参数和更低计算需求的同时，优于现有的轻量级视觉模型，适用于具有数据异质性和通信瓶颈的 FL 场景。