Dynamic diagnosis is desirable when medical tests are costly or
time-consuming. In this work, we use reinforcement learning (RL) to find a
dynamic policy that selects lab test panels sequentially based on previous
observations, ensuring accurate testing at a low cost. Clinical diagnostic data
are often highly imbalanced; therefore, we aim to maximize the $F_1$ score
instead of the error rate. However, optimizing the non-concave $F_1$ score is
not a classic RL problem, thus invalidates standard RL methods. To remedy this
issue, we develop a reward shaping approach, leveraging properties of the $F_1$
score and duality of policy optimization, to provably find the set of all
Pareto-optimal policies for budget-constrained $F_1$ score maximization. To
handle the combinatorially complex state space, we propose a Semi-Model-based
Deep Diagnosis Policy Optimization (SM-DDPO) framework that is compatible with
end-to-end training and online learning. SM-DDPO is tested on diverse clinical
tasks: ferritin abnormality detection, sepsis mortality prediction, and acute
kidney injury diagnosis. Experiments with real-world data validate that SM-DDPO
trains efficiently and identifies all Pareto-front solutions. Across all tasks,
SM-DDPO is able to achieve state-of-the-art diagnosis accuracy (in some cases
higher than conventional methods) with up to $85\%$ reduction in testing cost.
The code is available at
[this https URL].

本文介绍了一种利用增强学习和奖励设计方法优化实验测试序列，以最大化 F1 得分和降低成本的动态诊断策略。作者使用基于半模型深度诊断策略优化 (SM-DDPO) 的方法处理多种临床任务的数据，实现了高准确性和高成本效益