Training of modern large neural networks (NN) requires a combination of
parallelization strategies encompassing data, model, or optimizer sharding.
When strategies increase in complexity, it becomes necessary for partitioning
tools to be 1) expressive, allowing the composition of simpler strategies, and
2) predictable to estimate performance analytically. We present PartIR, our
design for a NN partitioning system. PartIR is focused on an incremental
approach to rewriting and is hardware-and-runtime agnostic. We present a simple
but powerful API for composing sharding strategies and a simulator to validate
them. The process is driven by high-level programmer-issued partitioning
tactics, which can be both manual and automatic. Importantly, the tactics are
specified separately from the model code, making them easy to change. We
evaluate PartIR on several different models to demonstrate its predictability,
expressibility, and ability to reach peak performance..

现代大型神经网络（NN）的训练需要结合数据、模型或优化器分片等并行化策略。当策略变得越来越复杂时，分区工具需要具备 1）表达能力，以允许复合简单的策略，2）可预测性，以便分析性能估计。我们提出了 PartIR，一种 NN 分区系统的设计。PartIR 着重于增量重写，并且不依赖于具体硬件和运行时环境。我们提供了一个简单但强大的 API 用于组合分片策略，并提供模拟器进行验证。该过程由高级程序员指定的分区策略驱动，可以手动或自动执行。重要的是，这些策略与模型代码分开指定，易于更改。我们在多个不同的模型上评估了 PartIR，以展示其可预测性、表达能力和达到峰值性能的能力。

PartIR: 机器学习 SPMD 划分策略的构建

PartIR: Composing SPMD Partitioning Strategies for Machine Learning

DevBots are automated tools that perform various tasks in order to support
software development. They are a growing trend and have been used in
repositories to automate repetitive tasks, as code generators, and as
collaborators in eliciting requirements and defining architectures. In this
study, we analyzed 24 articles to investigate the state of the art of using
DevBots in software development, trying to understand their characteristics,
identify use cases, learn the relationship between DevBots and conversational
software development, and discuss how prompt engineering can enable
collaboration between human developers and bots. Additionally, we identified a
gap to address by applying prompt engineering to collaborative API design
between human designers and DevBots and proposed an experiment to assess what
approach, between using Retrieval Augmented Generation or not, is more
suitable. Our conclusion is that DevBots can collaborate with human API
designers, but the two approaches have advantages and disadvantages.

DevBots 能够与人类 API 设计师合作，但是两种方法都有各自的优势和劣势。

DevBots 可以共同设计 API

DevBots can co-design APIs

Is it possible to design an universal API for federated learning using which
an ad-hoc group of data-holders (agents) collaborate with each other and
perform federated learning? Such an API would necessarily need to be
model-agnostic i.e. make no assumption about the model architecture being used
by the agents, and also cannot rely on having representative public data at
hand. Knowledge distillation (KD) is the obvious tool of choice to design such
protocols. However, surprisingly, we show that most natural KD-based federated
learning protocols have poor performance.
To investigate this, we propose a new theoretical framework, Federated Kernel
ridge regression, which can capture both model heterogeneity as well as data
heterogeneity. Our analysis shows that the degradation is largely due to a
fundamental limitation of knowledge distillation under data heterogeneity. We
further validate our framework by analyzing and designing new protocols based
on KD. Their performance on real world experiments using neural networks,
though still unsatisfactory, closely matches our theoretical predictions.

该研究考虑设计一个通用的 API 用于联邦学习，但发现大多数基于知识蒸馏的联邦学习协议表现不佳，因此提出了一个新的理论框架（联邦核岭回归）来解决模型和数据异构性的问题，并根据知识蒸馏设计了新的协议，通过实验验证了该框架的理论预测。

基于知识蒸馏的模型无关联合学习

Towards Model Agnostic Federated Learning Using Knowledge Distillation

Academia and industry have developed several platforms to support the popular
privacy-preserving distributed learning method -- Federated Learning (FL).
However, these platforms are complex to use and require a deep understanding of
FL, which imposes high barriers to entry for beginners, limits the productivity
of researchers, and compromises deployment efficiency. In this paper, we
propose the first low-code FL platform, EasyFL, to enable users with various
levels of expertise to experiment and prototype FL applications with little
coding. We achieve this goal while ensuring great flexibility and extensibility
for customization by unifying simple API design, modular design, and granular
training flow abstraction. With only a few lines of code, EasyFL empowers them
with many out-of-the-box functionalities to accelerate experimentation and
deployment. These practical functionalities are heterogeneity simulation,
comprehensive tracking, distributed training optimization, and seamless
deployment. They are proposed based on challenges identified in the proposed FL
life cycle. Compared with other platforms, EasyFL not only requires just three
lines of code (at least 10x lesser) to build a vanilla FL application but also
incurs lower training overhead. Besides, our evaluations demonstrate that
EasyFL expedites distributed training by 1.5x. It also improves the efficiency
of deployment. We believe that EasyFL will increase the productivity of
researchers and democratize FL to wider audiences.

本文介绍了一种名为 EasyFL 的低代码平台，可用于支持联邦学习方法。该平台不仅易于使用，而且具有模块化设计和简单的 API，可用于加速实验和部署。EasyFL 提供了多个功能，如模拟、全面跟踪、分布式培训优化和无缝部署，从而使联邦学习方法被更广泛地使用，并帮助提高研究人员的生产力。