Aiming at promoting the safe real-world deployment of Reinforcement Learning (RL), research on safe RL has made significant progress in recent years. However, most existing works in the literature still focus on the online setting where risky violations of the safety budget are likely to be incurred during training. Besides, in many real-world applications, the learned policy is required to respond to dynamically determined safety budgets (i.e., constraint threshold) in real time. In this paper, we target at the above real-time budget constraint problem under the offline setting, and propose Trajectory-based REal-time Budget Inference (TREBI) as a novel solution that approaches this problem from the perspective of trajectory distribution. Theoretically, we prove an error bound of the estimation on the episodic reward and cost under the offline setting and thus provide a performance guarantee for TREBI. Empirical results on a wide range of simulation tasks and a real-world large-scale advertising application demonstrate the capability of TREBI in solving real-time budget constraint problems under offline settings.

本文从轨迹分布角度提出了基于Trajectory-based REal-time Budget Inference(TREBI)的解决方案，以解决离线情况下真实时间预算约束问题，理论上证明了收益和成本估计的误差界限，为TREBI提供性能保证。实验结果在广告应用中证明了TREBI在解决离线环境下的实时预算约束问题的能力。

具有实时预算约束的安全离线强化学习