Artificial intelligence (AI) solutions that automatically extract information
from digital histology images have shown great promise for improving
pathological diagnosis. Prior to routine use, it is important to evaluate their
predictive performance and obtain regulatory approval. This assessment requires
appropriate test datasets. However, compiling such datasets is challenging and
specific recommendations are missing.
A committee of various stakeholders, including commercial AI developers,
pathologists, and researchers, discussed key aspects and conducted extensive
literature reviews on test datasets in pathology. Here, we summarize the
results and derive general recommendations for the collection of test datasets.
We address several questions: Which and how many images are needed? How to
deal with low-prevalence subsets? How can potential bias be detected? How
should datasets be reported? What are the regulatory requirements in different
countries?
The recommendations are intended to help AI developers demonstrate the
utility of their products and to help regulatory agencies and end users verify
reported performance measures. Further research is needed to formulate criteria
for sufficiently representative test datasets so that AI solutions can operate
with less user intervention and better support diagnostic workflows in the
future.

人工智能在数字组织学图像中的自动信息提取已被证明可以改善病理诊断。然而，在正式使用前，需要评估其预测性能并获得监管机构的认可，而这需要恰当的测试数据集，本文总结了病理测试数据集的一般建议，旨在帮助人工智能开发者证明其产品的实用性，并帮助监管机构和最终用户验证所报导的性能指标。