Automated anesthesia promises to enable more precise and personalized
anesthetic administration and free anesthesiologists from repetitive tasks,
allowing them to focus on the most critical aspects of a patient's surgical
care. Current research has typically focused on creating simulated environments
from which agents can learn. These approaches have demonstrated good
experimental results, but are still far from clinical application. In this
paper, Policy Constraint Q-Learning (PCQL), a data-driven reinforcement
learning algorithm for solving the problem of learning anesthesia strategies on
real clinical datasets, is proposed. Conservative Q-Learning was first
introduced to alleviate the problem of Q function overestimation in an offline
context. A policy constraint term is added to agent training to keep the policy
distribution of the agent and the anesthesiologist consistent to ensure safer
decisions made by the agent in anesthesia scenarios. The effectiveness of PCQL
was validated by extensive experiments on a real clinical anesthesia dataset.
Experimental results show that PCQL is predicted to achieve higher gains than
the baseline approach while maintaining good agreement with the reference dose
given by the anesthesiologist, using less total dose, and being more responsive
to the patient's vital signs. In addition, the confidence intervals of the
agent were investigated, which were able to cover most of the clinical
decisions of the anesthesiologist. Finally, an interpretable method, SHAP, was
used to analyze the contributing components of the model predictions to
increase the transparency of the model.

提出了基于数据驱动增强学习算法的策略约束 Q-Learning (PCQL) 以解决学习麻醉策略在真实临床数据集上的问题，并通过实验证实了 PCQL 相对于基线方法的性能优势，在更少的总剂量下保持和麻醉师参考剂量的一致性，对病人的生命体征更加敏感，并使用 SHAP 进行了模型预测的解释分析。