Machine learning (ML) models can fail in unexpected ways in the real world,
but not all model failures are equal. With finite time and resources, ML
practitioners are forced to prioritize their model debugging and improvement
efforts. Through interviews with 13 ML practitioners at Apple, we found that
practitioners construct small targeted test sets to estimate an error's nature,
scope, and impact on users. We built on this insight in a case study with
machine translation models, and developed Angler, an interactive visual
analytics tool to help practitioners prioritize model improvements. In a user
study with 7 machine translation experts, we used Angler to understand
prioritization practices when the input space is infinite, and obtaining
reliable signals of model quality is expensive. Our study revealed that
participants could form more interesting and user-focused hypotheses for
prioritization by analyzing quantitative summary statistics and qualitatively
assessing data by reading sentences.

通过采访苹果的 13 名机器学习从业者，研究人员发现他们构建小的有针对性的测试集来评估一个错误的性质、范围以及对用户的影响。研究者开发了一种交互式的可视化分析工具 ——Angler，以帮助从业者优先改进模型。研究发现，参与者通过分析定量摘要统计信息和定性评估数据读取句子，可以形成更有趣和用户关注的假设来优化模型。