Recently, self-learning methods based on user satisfaction metrics and
contextual bandits have shown promising results to enable consistent
improvements in conversational AI systems. However, directly targeting such
metrics by off-policy bandit learning objectives often increases the risk of
making abrupt policy changes that break the current user experience. In this
study, we introduce a scalable framework for supporting fine-grained
exploration targets for individual domains via user-defined constraints. For
example, we may want to ensure fewer policy deviations in business-critical
domains such as shopping, while allocating more exploration budget to domains
such as music. Furthermore, we present a novel meta-gradient learning approach
that is scalable and practical to address this problem. The proposed method
adjusts constraint violation penalty terms adaptively through a meta objective
that encourages balanced constraint satisfaction across domains. We conduct
extensive experiments using data from a real-world conversational AI on a set
of realistic constraint benchmarks. Based on the experimental results, we
demonstrate that the proposed approach is capable of achieving the best balance
between the policy value and constraint satisfaction rate.

本研究提出了一种可扩展的框架处理个别领域的细粒度探索目标，通过用户定义的限制来确保商业关键领域的政策偏差较少，并将更多的探索预算分配到音乐等领域，提出的方法通过一个元目标调整约束违规惩罚项的自适应，鼓励领域间平衡的约束满足，实验结果表明该方法在政策价值和约束满足率之间实现了最佳平衡。