We introduce a new architecture for Distributional Reinforcement Learning (DistRL) that models return distributions using normalizing flows. This approach enables flexible, unbounded support for return distributions, in contrast to categorical approaches like C51 that rely on fixed or bounded representations. It also offers richer modeling capacity to capture multi-modality, skewness, and tail behavior than quantile based approaches. Our method is significantly more parameter-efficient than categorical approaches. Standard metrics used to train existing models like KL divergence or Wasserstein distance either are scale insensitive or have biased sample gradients, especially when return supports do not overlap. To address this, we propose a novel surrogate for the Cram\`er distance, that is geometry-aware and computable directly from the return distribution's PDF, avoiding the costly CDF computation. We test our model on the ATARI-5 sub-benchmark and show that our approach outperforms PDF based models while remaining competitive with quantile based methods.

本文提出了一种新的分布式强化学习架构，利用归一化流模型返回分布，相较于传统的分类方法如C51，具有灵活的无限支持能力。研究显示该方法在参数效率上显著优于分类方法，并通过几何感知的新代理Cramér距离优化现有模型度量，使得在不重叠的返回支持下表现更佳。我们在ATARI-5子基准测试中验证了该模型，相较于基于PDF的方法表现优越，同时与基于分位数的方法竞争力强。 

面向无限制和几何感知的分布式强化学习的流模型