Large transformers have achieved state-of-the-art performance across many tasks. Most open-source libraries on scaling transformers focus on improving training or inference with better parallelization. In this wo