Over the past few years, research has witnessed the advancement of deep
learning models trained on large datasets, some even encompassing millions of
examples. While these impressive performance on their hidden test sets, they
often underperform when assessed on external datasets. Recognizing the critical
role of generalization in medical AI development, many prestigious journals now
require reporting results both on the local hidden test set as well as on
external datasets before considering a study for publication. Effectively, the
field of medical AI has transitioned from the traditional usage of a single
dataset that is split into train and test to a more comprehensive framework
using multiple datasets, some of which are used for model development (source
domain) and others for testing (target domains). However, this new experimental
setting does not necessarily resolve the challenge of generalization. This is
because of the variability encountered in intended use and specificities across
hospital cultures making the idea of universally generalizable systems a myth.
On the other hand, the systematic, and a fortiori recurrent re-calibration, of
models at the individual hospital level, although ideal, may be overoptimistic
given the legal, regulatory and technical challenges that are involved.
Re-calibration using transfer learning may not even be possible in some
instances where reference labels of target domains are not available. In this
perspective we establish a hierarchical three-level scale system reflecting the
generalization level of a medical AI algorithm. This scale better reflects the
diversity of real-world medical scenarios per which target domain data for
re-calibration of models may or not be available and if it is, may or not have
reference labels systematically available.

医学人工智能算法的泛化水平可以通过建立一个分层三级评估系统来反映，该系统更好地反映了真实医疗情境的多样性，其中用于重新校准模型的目标领域数据可能可用也可能不可用，并且如果可用的话，可能或可能不会系统地提供参考标签。