Federated Learning (FL) stands out as a widely adopted protocol facilitating
the training of Machine Learning (ML) models while maintaining decentralized
data. However, challenges arise when dealing with a heterogeneous set of
participating devices, causing delays in the training process, particularly
among devices with limited resources. Moreover, the task of training ML models
with a vast number of parameters demands computing and memory resources beyond
the capabilities of small devices, such as mobile and Internet of Things (IoT)
devices. To address these issues, techniques like Parallel Split Learning (SL)
have been introduced, allowing multiple resource-constrained devices to
actively participate in collaborative training processes with assistance from
resourceful compute nodes. Nonetheless, a drawback of Parallel SL is the
substantial memory allocation required at the compute nodes, for instance
training VGG-19 with 100 participants needs 80 GB. In this paper, we introduce
Multihop Parallel SL (MP-SL), a modular and extensible ML as a Service (MLaaS)
framework designed to facilitate the involvement of resource-constrained
devices in collaborative and distributed ML model training. Notably, to
alleviate memory demands per compute node, MP-SL supports multihop Parallel
SL-based training. This involves splitting the model into multiple parts and
utilizing multiple compute nodes in a pipelined manner. Extensive
experimentation validates MP-SL's capability to handle system heterogeneity,
demonstrating that the multihop configuration proves more efficient than
horizontally scaled one-hop Parallel SL setups, especially in scenarios
involving more cost-effective compute nodes.

多跳并行分裂学习（MP-SL）是一个模块化且可拓展的机器学习作为服务（MLaaS）框架，旨在促进资源受限设备参与分布式协作的机器学习模型训练，通过将模型分割为多个部分并利用多个计算节点以流水线方式进行训练，以减轻计算节点的内存需求，具有处理系统异构性的能力，尤其在涉及成本更低效的计算节点的场景中比水平扩展的单跳并行分裂学习设置更高效。