Multi-agent reinforcement learning (MARL) is a powerful technology to
construct interactive artificial intelligent systems in various applications
such as multi-robot control and self-driving cars. Unlike supervised model or
single-agent reinforcement learning, which actively exploits network pruning,
it is obscure that how pruning will work in multi-agent reinforcement learning
with its cooperative and interactive characteristics. \par In this paper, we
present a real-time sparse training acceleration system named LearningGroup,
which adopts network pruning on the training of MARL for the first time with an
algorithm/architecture co-design approach. We create sparsity using a weight
grouping algorithm and propose on-chip sparse data encoding loop (OSEL) that
enables fast encoding with efficient implementation. Based on the OSEL's
encoding format, LearningGroup performs efficient weight compression and
computation workload allocation to multiple cores, where each core handles
multiple sparse rows of the weight matrix simultaneously with vector processing
units. As a result, LearningGroup system minimizes the cycle time and memory
footprint for sparse data generation up to 5.72x and 6.81x. Its FPGA
accelerator shows 257.40-3629.48 GFLOPS throughput and 7.10-100.12 GFLOPS/W
energy efficiency for various conditions in MARL, which are 7.13x higher and
12.43x more energy efficient than Nvidia Titan RTX GPU, thanks to the fully
on-chip training and highly optimized dataflow/data format provided by FPGA.
Most importantly, the accelerator shows speedup up to 12.52x for processing
sparse data over the dense case, which is the highest among state-of-the-art
sparse training accelerators.

本研究提出了一种名为 LearningGroup 的实时稀疏训练加速系统，采用针对多智能体强化学习的网络剪枝，并结合算法 / 架构协同设计。该系统在 FPGA 加速器上实现全面的芯片内训练和高度优化的数据流 / 数据格式，以最小化稀疏数据产生的周期时间和内存占用，并在处理稀疏数据的效率上显示出 12.52 倍的加速。