Much machine learning research progress is based on developing models and
evaluating them on a benchmark dataset (e.g., ImageNet for images). However,
applying such benchmark-successful methods to real-world data often does not
work as expected. This is particularly the case for biological data where we
expect variability at multiple time and spatial scales. In this work, we are
using grain data and the goal is to detect diseases and damages. Pink fusarium,
skinned grains, and other diseases and damages are key factors in setting the
price of grains or excluding dangerous grains from food production. Apart from
challenges stemming from differences of the data from the standard toy
datasets, we also present challenges that need to be overcome when explaining
deep learning models. For example, explainability methods have many
hyperparameters that can give different results, and the ones published in the
papers do not work on dissimilar images. Other challenges are more general:
problems with visualization of the explanations and their comparison since the
magnitudes of their values differ from method to method. An open fundamental
question also is: How to evaluate explanations? It is a non-trivial task
because the "ground truth" is usually missing or ill-defined. Also, human
annotators may create what they think is an explanation of the task at hand,
yet the machine learning model might solve it in a different and perhaps
counter-intuitive way. We discuss several of these challenges and evaluate
various post-hoc explainability methods on grain data. We focus on robustness,
quality of explanations, and similarity to particular "ground truth"
annotations made by experts. The goal is to find the methods that overall
perform well and could be used in this challenging task. We hope the proposed
pipeline will be used as a framework for evaluating explainability methods in
specific use cases.

通过评估基准数据集上的模型并将其应用于真实世界数据，本研究探讨了在生物数据领域应用机器学习模型的挑战以及解释性方法的评估，以检测谷物疾病和损坏。

应对生物数据变异的深度学习模型解释挑战

Challenges in explaining deep learning models for data with biological  variation

Post-hoc explainability methods aim to clarify predictions of black-box
machine learning models. However, it is still largely unclear how well users
comprehend the provided explanations and whether these increase the users
ability to predict the model behavior. We approach this question by conducting
a user study to evaluate comprehensibility and predictability in two widely
used tools: LIME and SHAP. Moreover, we investigate the effect of
counterfactual explanations and misclassifications on users ability to
understand and predict the model behavior. We find that the comprehensibility
of SHAP is significantly reduced when explanations are provided for samples
near a model's decision boundary. Furthermore, we find that counterfactual
explanations and misclassifications can significantly increase the users
understanding of how a machine learning model is making decisions. Based on our
findings, we also derive design recommendations for future post-hoc
explainability methods with increased comprehensibility and predictability.

评估 LIME 和 SHAP 两种常用工具的用户可理解性和可预测性，发现 SHAP 对于接近模型决策边界的样本提供的解释可理解性显著降低。此外，发现反事实解释和错误分类可以显著增加用户对机器学习模型决策的理解。根据研究结果，提出为未来的事后解释方法提供增加可理解性和可预测性的设计建议。