Oobleck enables resilient distributed training of large DNN models with
guaranteed fault tolerance. It takes a planning-execution co-design approach,
where it first generates a set of heterogeneous pipeline templates and
instantiates at least $f+1$ logically equivalent pipeline replicas to tolerate
any $f$ simultaneous failures. During execution, it relies on
already-replicated model states across the replicas to provide fast recovery.
Oobleck provably guarantees that some combination of the initially created
pipeline templates can be used to cover all available resources after $f$ or
fewer simultaneous failures, thereby avoiding resource idling at all times.
Evaluation on large DNN models with billions of parameters shows that Oobleck
provides consistently high throughput, and it outperforms state-of-the-art
fault tolerance solutions like Bamboo and Varuna by up to $13.9x$.

Oobleck 采用规划执行的共同设计方法，通过生成一组异构的流水线模板，并实例化至少 f+1 个逻辑上等效的流水线副本以容忍 f 个同时故障，依赖于副本之间已复制的模型状态快速恢复，并保证在 f 个或更少的同时故障后，初始创建的流水线模板可以覆盖所有可用资源，从而始终避免资源闲置。在具有数十亿个参数的大规模 DNN 模型上的评估表明，Oobleck 提供了一致高吞吐量，并且比像 Bamboo 和 Varuna 这样的最新故障容错解决方案表现优越，提升了 13.9 倍。