Estimating conditional average dose responses (CADR) is an important but
challenging problem. Estimators must correctly model the potentially complex
relationships between covariates, interventions, doses, and outcomes. In recent
years, the machine learning community has shown great interest in developing
tailored CADR estimators that target specific challenges. Their performance is
typically evaluated against other methods on (semi-) synthetic benchmark
datasets. Our paper analyses this practice and shows that using popular
benchmark datasets without further analysis is insufficient to judge model
performance. Established benchmarks entail multiple challenges, whose impacts
must be disentangled. Therefore, we propose a novel decomposition scheme that
allows the evaluation of the impact of five distinct components contributing to
CADR estimator performance. We apply this scheme to eight popular CADR
estimators on four widely-used benchmark datasets, running nearly 1,500
individual experiments. Our results reveal that most established benchmarks are
challenging for reasons different from their creators' claims. Notably,
confounding, the key challenge tackled by most estimators, is not an issue in
any of the considered datasets. We discuss the major implications of our
findings and present directions for future research.

我们提出了一种新的分解方案，用于评估影响 CADR 估计器性能的五个不同组成部分的影响。我们运行了近 1500 个独立实验，将这个方案应用于四个广泛使用的基准数据集上的八个流行的 CADR 估计器。我们的结果揭示了大多数现有的基准数据集的挑战不同于其创建者所声称的挑战，同时发现干扰并不是任何考虑的数据集中的问题。我们讨论了我们发现的主要影响，并提出了未来研究的方向。