Off-Policy reinforcement learning has been a driving force for the state-of-the-art conversational AIs leading to more natural humanagent interactions and improving the user satisfaction for goal-oriented agents. However, in large-scale commercial settings, it is often challenging to balance between policy improvements and experience continuity on the broad spectrum of applications handled by such system. In the literature, off-policy evaluation and guard-railing on aggregate statistics has been commonly used to address this problem. In this paper, we propose a method for curating and leveraging high-precision samples sourced from historical regression incident reports to validate, safe-guard, and improve policies prior to the online deployment. We conducted extensive experiments using data from a real-world conversational system and actual regression incidents. The proposed method is currently deployed in our production system to protect customers against broken experiences and enable long-term policy improvements.

本文提出了一种基于历史回归事故报告的高精度数据样本的培育和利用方法，以在在线部署之前验证、保护并改进政策，解决Off-Policy强化学习在大规模商业设置中难以平衡政策改进和经验连续性的问题，并提高了对话系统的用户满意度。

自学对话系统中缺陷行为的可扩展和安全修复