In Federated Learning (FL), the limited accessibility of data from diverse
locations and user types poses a significant challenge due to restricted user
participation. Expanding client access and diversifying data enhance models by
incorporating diverse perspectives, thereby enhancing adaptability. However,
challenges arise in dynamic and mobile environments where certain devices may
become inaccessible as FL clients, impacting data availability and client
selection methods. To address this, we propose an On-Demand solution, deploying
new clients using Docker Containers on-the-fly. Our On-Demand solution,
employing Deep Reinforcement Learning (DRL), targets client availability and
selection, while considering data shifts, and container deployment
complexities. It employs an autonomous end-to-end solution for handling model
deployment and client selection. The DRL strategy uses a Markov Decision
Process (MDP) framework, with a Master Learner and a Joiner Learner. The
designed cost functions represent the complexity of the dynamic client
deployment and selection. Simulated tests show that our architecture can easily
adjust to changes in the environment and respond to On-Demand requests. This
underscores its ability to improve client availability, capability, accuracy,
and learning efficiency, surpassing heuristic and tabular reinforcement
learning solutions.

我们提出了一个利用深度强化学习技术的按需解决方案，通过在运行时使用 Docker 容器部署新的客户端，以解决动态和移动环境中的数据可用性和客户端选择的挑战。我们的架构能够适应环境变化，并对按需请求做出响应，从而提高客户端的可用性、能力、准确性和学习效率，超越启发式和表格型强化学习解决方案。

基于深度强化学习的联邦学习中的按需模型和客户端部署

On-Demand Model and Client Deployment in Federated Learning with Deep  Reinforcement Learning

While federated learning has shown strong results in optimizing a machine
learning model without direct access to the original data, its performance may
be hindered by intermittent client availability which slows down the
convergence and biases the final learned model. There are significant
challenges to achieve both stable and bias-free training under arbitrary client
availability. To address these challenges, we propose a framework named
Federated Graph-based Sampling (FedGS), to stabilize the global model update
and mitigate the long-term bias given arbitrary client availability
simultaneously. First, we model the data correlations of clients with a
Data-Distribution-Dependency Graph (3DG) that helps keep the sampled clients
data apart from each other, which is theoretically shown to improve the
approximation to the optimal model update. Second, constrained by the
far-distance in data distribution of the sampled clients, we further minimize
the variance of the numbers of times that the clients are sampled, to mitigate
long-term bias. To validate the effectiveness of FedGS, we conduct experiments
on three datasets under a comprehensive set of seven client availability modes.
Our experimental results confirm FedGS's advantage in both enabling a fair
client-sampling scheme and improving the model performance under arbitrary
client availability. Our code is available at
https://github.com/WwZzz/FedGS.

提出一种名为 FedGS 的框架，通过构建数据依赖图、限制采样次数等方法稳定了全局模型更新，解决了联邦学习中由于客户端可用性不稳定带来的模型偏差问题。实验结果验证了 FedGS 在实现公平客户端采样以及提升模型性能方面的优势。