User simulators are essential for training reinforcement learning (RL) based
dialog models. The performance of the simulator directly impacts the RL policy.
However, building a good user simulator that models real user behaviors is
challenging. We propose a method of standardizing user simulator building that
can be used by the community to compare dialog system quality using the same
set of user simulators fairly. We present implementations of six user
simulators trained with different dialog planning and generation methods. We
then calculate a set of automatic metrics to evaluate the quality of these
simulators both directly and indirectly. We also ask human users to assess the
simulators directly and indirectly by rating the simulated dialogs and
interacting with the trained systems. This paper presents a comprehensive
evaluation framework for user simulator study and provides a better
understanding of the pros and cons of different user simulators, as well as
their impacts on the trained systems.

我们提出一种标准化用户模拟器构建的方法，用于评估对话系统质量。我们使用不同的对话规划和生成方法训练六个用户模拟器，并计算一组自动指标来评估这些模拟器的质量。此外，通过让人类用户评估模拟器并与训练系统交互，间接地和直接地评估了这些模拟器。本文提供了一个全面的用户模拟器研究评估框架，并更好地理解了不同用户模拟器的优缺点及其对训练系统的影响。

如何构建用户模拟器以训练基于强化学习的对话系统

How to Build User Simulators to Train RL-based Dialog Systems

Simulation and bisimulation metrics for stochastic systems provide a
quantitative generalization of the classical simulation and bisimulation
relations. These metrics capture the similarity of states with respect to
quantitative specifications written in the quantitative {\mu}-calculus and
related probabilistic logics. We first show that the metrics provide a bound
for the difference in long-run average and discounted average behavior across
states, indicating that the metrics can be used both in system verification,
and in performance evaluation. For turn-based games and MDPs, we provide a
polynomial-time algorithm for the computation of the one-step metric distance
between states. The algorithm is based on linear programming; it improves on
the previous known exponential-time algorithm based on a reduction to the
theory of reals. We then present PSPACE algorithms for both the decision
problem and the problem of approximating the metric distance between two
states, matching the best known algorithms for Markov chains. For the
bisimulation kernel of the metric our algorithm works in time O(n^4) for both
turn-based games and MDPs; improving the previously best known O(n^9\cdot
log(n)) time algorithm for MDPs. For a concurrent game G, we show that
computing the exact distance between states is at least as hard as computing
the value of concurrent reachability games and the square-root-sum problem in
computational geometry. We show that checking whether the metric distance is
bounded by a rational r, can be done via a reduction to the theory of real
closed fields, involving a formula with three quantifier alternations, yielding
O(|G|^O(|G|^5)) time complexity, improving the previously known reduction,
which yielded O(|G|^O(|G|^7)) time complexity. These algorithms can be iterated
to approximate the metrics using binary search.

基于模拟和双模拟的指标可以用于系统验证和性能评估，适用于定量的 mu - 演算和相关概率逻辑，对于马尔可夫链，我们提供了一个 PSPACE 算法，以匹配最佳算法，并且这些算法可以通过二分搜索来逼近指标。