Deferring systems extend supervised Machine Learning (ML) models with the
possibility to defer predictions to human experts. However, evaluating the
impact of a deferring strategy on system accuracy is still an overlooked area.
This paper fills this gap by evaluating deferring systems through a causal
lens. We link the potential outcomes framework for causal inference with
deferring systems. This allows us to identify the causal impact of the
deferring strategy on predictive accuracy. We distinguish two scenarios. In the
first one, we can access both the human and the ML model predictions for the
deferred instances. In such a case, we can identify the individual causal
effects for deferred instances and aggregates of them. In the second scenario,
only human predictions are available for the deferred instances. In this case,
we can resort to regression discontinuity design to estimate a local causal
effect. We empirically evaluate our approach on synthetic and real datasets for
seven deferring systems from the literature.

通过因果推断的方法，评估延迟系统对预测准确度的因果影响，区别两种场景，一种是能够接触到人类和机器学习模型的预测结果，另一种是只有人类预测结果可用，通过这种方法在合成和真实数据集上对七种延迟系统进行了实证评估。