The Large Scale Visual Recognition Challenge based on the well-known Imagenet dataset catalyzed an intense flurry of progress in computer vision. Benchmark tasks have propelled other sub-fields of machine learning forward at an equally impressive pace, but in healthcare it has primarily been image processing tasks, such as in dermatology and radiology, that have experienced similar benchmark-driven progress. In the present study, we performed a comprehensive review of benchmarks in medical machine learning for structured data, identifying one based on the Medical Information Mart for Intensive Care (MIMIC-III) that allows the first direct comparison of predictive performance and thus the evaluation of progress on four clinical prediction tasks: mortality, length of stay, phenotyping, and patient decompensation. We find that little meaningful progress has been made over a 3 year period on these tasks, despite significant community engagement. Through our meta-analysis, we find that the performance of deep recurrent models is only superior to logistic regression on certain tasks. We conclude with a synthesis of these results, possible explanations, and a list of desirable qualities for future benchmarks in medical machine learning.

综合评估医疗机器学习的基准，提出一种基于医学信息志(MIMIC-III)的基准，让我们可以直接比较预测性能，并评估生存率、住院时间、表型和患者恶化任务的进展。我们发现，在这些任务上，尽管社区参与度高，但在过去的三年里，几乎没有真正显著的进展。通过我们的元分析，我们发现，深度递归模型的性能仅在某些任务上优于逻辑回归。最后综合这些结果，提出未来医学机器学习基准所需的理想特性。 

评估机器学习在纵向电子医疗数据上的进展