Deep Reinforcement Learning (RL) has been demonstrated to yield capable
agents and control policies in several domains but is commonly plagued by
prohibitively long training times. Additionally, in the case of continuous
control problems, the applicability of learned policies on real-world embedded
devices is limited due to the lack of real-time guarantees and portability of
existing deep learning libraries. To address these challenges, we present
BackpropTools, a dependency-free, header-only, pure C++ library for deep
supervised and reinforcement learning. Leveraging the template meta-programming
capabilities of recent C++ standards, we provide composable components that can
be tightly integrated by the compiler. Its novel architecture allows
BackpropTools to be used seamlessly on a heterogeneous set of platforms, from
HPC clusters over workstations and laptops to smartphones, smartwatches, and
microcontrollers. Specifically, due to the tight integration of the RL
algorithms with simulation environments, BackpropTools can solve popular RL
problems like the Pendulum-v1 swing-up about 7 to 15 times faster in terms of
wall-clock training time compared to other popular RL frameworks when using
TD3. We also provide a low-overhead and parallelized interface to the MuJoCo
simulator, showing that our PPO implementation achieves state of the art
returns in the Ant-v4 environment while achieving a 25 to 30 percent faster
wall-clock training time. Finally, we also benchmark the policy inference on a
diverse set of microcontrollers and show that in most cases our optimized
inference implementation is much faster than even the manufacturer's DSP
libraries. To the best of our knowledge, BackpropTools enables the first-ever
demonstration of training a deep RL algorithm directly on a microcontroller,
giving rise to the field of Tiny Reinforcement Learning (TinyRL). Project page:
this https URL

BackpropTools 是一个用于深度监督和强化学习的 C++ 库，能够在各种平台上高效地工作，其 RL 算法与模拟环境的紧密集成使得在微控制器上直接训练深度 RL 算法成为可能，这使得 “微型强化学习 (TinyRL)” 领域初次呈现。