A growing body of research runs human subject evaluations to study whether
providing users with explanations of machine learning models can help them with
practical real-world use cases. However, running user studies is challenging
and costly, and consequently each study typically only evaluates a limited
number of different settings, e.g., studies often only evaluate a few
arbitrarily selected explanation methods. To address these challenges and aid
user study design, we introduce Use-Case-Grounded Simulated Evaluations
(SimEvals). SimEvals involve training algorithmic agents that take as input the
information content (such as model explanations) that would be presented to
each participant in a human subject study, to predict answers to the use case
of interest. The algorithmic agent's test set accuracy provides a measure of
the predictiveness of the information content for the downstream use case. We
run a comprehensive evaluation on three real-world use cases (forward
simulation, model debugging, and counterfactual reasoning) to demonstrate that
Simevals can effectively identify which explanation methods will help humans
for each use case. These results provide evidence that SimEvals can be used to
efficiently screen an important set of user study design decisions, e.g.
selecting which explanations should be presented to the user, before running a
potentially costly user study.

该论文提出了使用基于用例的模拟评估（SimEvals）来评估机器学习模型中的解释方法是否有助于实际应用，并且通过三个真实世界的用例进行了综合评估，以证明 SimEvals 可以有效地确定哪些解释方法适用于每个用例。