We present Inferflow, an efficient and highly configurable inference engine
for large language models (LLMs). With Inferflow, users can serve most of the
common transformer models by simply modifying some lines in corresponding
configuration files, without writing a single line of source code. Compared
with most existing inference engines, Inferflow has some key features. First,
by implementing a modular framework of atomic build-blocks and technologies,
Inferflow is compositionally generalizable to new models. Second, 3.5-bit
quantization is introduced in Inferflow as a tradeoff between 3-bit and 4-bit
quantization. Third, hybrid model partitioning for multi-GPU inference is
introduced in Inferflow to better balance inference speed and throughput than
the existing partition-by-layer and partition-by-tensor strategies.

Inferflow 是一个有效且高度可配置的推理引擎，适用于大规模语言模型（LLMs）。通过修改相关配置文件中的几行代码，用户可以简单地为大多数常见的 Transformer 模型提供服务，而无需编写源代码。相比其他推理引擎，Inferflow 具有一些关键特性：首先，通过实现原子建模块和技术的模块化框架，Inferflow 可以普遍适用于新模型；其次，引入了 3.5 位量化作为 3 位量化和 4 位量化之间的折中；第三，Inferflow 引入了混合模型分区以进行多 GPU 推理，以更好地平衡推理速度和吞吐量，超过了现有的按层分区和按张量分区策略。

Inferflow：一个高效和高度可配置的大语言模型推理引擎

Inferflow: an Efficient and Highly Configurable Inference Engine for  Large Language Models

Hierarchical federated learning (HFL) has demonstrated promising scalability
advantages over the traditional "star-topology" architecture-based federated
learning (FL). However, HFL still imposes significant computation,
communication, and storage burdens on the edge, especially when training a
large-scale model over resource-constrained Internet of Things (IoT) devices.
In this paper, we propose hierarchical independent submodel training (HIST), a
new FL methodology that aims to address these issues in hierarchical settings.
The key idea behind HIST is a hierarchical version of model partitioning, where
we partition the global model into disjoint submodels in each round, and
distribute them across different cells, so that each cell is responsible for
training only one partition of the full model. This enables each client to save
computation/storage costs while alleviating the communication loads throughout
the hierarchy. We characterize the convergence behavior of HIST for non-convex
loss functions under mild assumptions, showing the impact of several attributes
(e.g., number of cells, local and global aggregation frequency) on the
performance-efficiency tradeoff. Finally, through numerical experiments, we
verify that HIST is able to save communication costs by a wide margin while
achieving the same target testing accuracy.

我们提出了一种新的分层联邦学习方法 - 分层独立子模型训练（HIST），它通过将全局模型划分为不相交的子模型，并将它们分布在不同的细胞中来解决分层环境下的计算、存储和通信负担问题。我们证明了 HIST 对于非凸损失函数的收敛行为，并展示了几个属性（例如细胞的数量、本地和全局聚合频率）对性能效率权衡的影响。通过数值实验，我们验证了 HIST 能够大幅节省通信成本，并实现相同的目标测试准确率。