BriefGPT.xyz
Jan, 2024
T3: 透明化追踪与触发技术用于计算和集合的细粒度叠加
T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives
HTML
PDF
Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, Matthew D. Sinclair
TL;DR
T3是一种通过硬件-软件协同设计,透明地重叠序列化的通信和计算,并最小化资源争用的方法,可对Transformer模型进行加速,减少数据移动。
Abstract
large language models
increasingly rely on
distributed techniques
for their training and inference. These techniques require communication across devices which can reduce scaling efficiency as the number of devic
→