Pipeline parallelism has been widely explored, but most existing schedules
lack a systematic methodology. In this paper, we propose a framework to
decompose pipeline schedules as repeating a building block and we show that the
lifespan of the building block decides the peak activation memory of the
pipeline schedule. Guided by the observations, we find that almost all existing
pipeline schedules, to the best of our knowledge, are memory inefficient. To
address this, we introduce a family of memory efficient building blocks with
controllable activation memory, which can reduce the peak activation memory to
1/2 of 1F1B without sacrificing efficiency, and even to 1/3 with comparable
throughput. We can also achieve almost zero pipeline bubbles while maintaining
the same activation memory as 1F1B. Our evaluations demonstrate that in pure
pipeline parallelism settings, our methods outperform 1F1B by from 7% to 55% in
terms of throughput. When employing a grid search over hybrid parallelism
hyperparameters in practical scenarios, our proposed methods demonstrate a 16%
throughput improvement over the 1F1B baseline for large language models.

通过重复构建块的方式，将流水线调度分解为生命周期的概念，并发现现有调度方案在内存利用方面存在问题。为了解决这个问题，引入了一系列具有可控激活内存的高效构建块，可以在不影响效率的情况下将峰值激活内存减少到 1F1B 的 1/2 甚至 1/3，同时几乎没有流水线气泡，以及在吞吐量方面表现优于 1F1B。在实践场景中对混合并行化超参数进行网格搜索时，相较于 1F1B 基准，我们提出的方法在大型语言模型上实现了 16% 的吞吐量提升。

可控内存的管道并行

Pipeline Parallelism with Controllable Memory

Stackelberg Security Games are often used to model strategic interactions in
high-stakes security settings. The majority of existing models focus on
single-defender settings where a single entity assumes command of all security
assets. However, many realistic scenarios feature multiple heterogeneous
defenders with their own interests and priorities embedded in a more complex
system. Furthermore, defenders rarely choose targets to protect. Instead, they
have a multitude of defensive resources or schedules at its disposal, each with
different protective capabilities. In this paper, we study security games
featuring multiple defenders and schedules simultaneously. We show that unlike
prior work on multi-defender security games, the introduction of schedules can
cause non-existence of equilibrium even under rather restricted environments.
We prove that under the mild restriction that any subset of a schedule is also
a schedule, non-existence of equilibrium is not only avoided, but can be
computed in polynomial time in games with two defenders. Under additional
assumptions, our algorithm can be extended to games with more than two
defenders and its computation scaled up in special classes of games with
compactly represented schedules such as those used in patrolling applications.
Experimental results suggest that our methods scale gracefully with game size,
making our algorithms amongst the few that can tackle multiple heterogeneous
defenders.

研究了包含多个卫士和时间表的安全游戏，发现时间表的引入可能导致平衡不存在，但在特定条件下，能够在多个卫士的游戏中以多项式时间计算避免平衡不存在的问题。