As the use of machine learning (ML) models in product development and data-driven decision-making processes became pervasive in many domains, people's focus on building a well-performing model has increasingly shifted to understanding how their model works. While scholarly interest in model interpretability has grown rapidly in research communities like HCI, ML, and beyond, little is known about how practitioners perceive and aim to provide interpretability in the context of their existing workflows. This lack of understanding of interpretability as practiced may prevent interpretability research from addressing important needs, or lead to unrealistic solutions. To bridge this gap, we conducted 22 semi-structured interviews with industry practitioners to understand how they conceive of and design for interpretability while they plan, build, and use their models. Based on a qualitative analysis of our results, we differentiate interpretability roles, processes, goals and strategies as they exist within organizations making heavy use of ML models. The characterization of interpretability work that emerges from our analysis suggests that model interpretability frequently involves cooperation and mental model comparison between people in different roles, often aimed at building trust not only between people and models but also between people within the organization. We present implications for design that discuss gaps between the interpretability challenges that practitioners face in their practice and approaches proposed in the literature, highlighting possible research directions that can better address real-world needs.

我们对22位业内从业人员的半结构化访谈结果进行了定性分析，提出了在大量使用机器学习模型的组织中存在的解释性工作的特征，这表明模型可解释性通常涉及不同角色的人员之间的合作和心理模型比较，旨在建立人员和模型之间以及组织内部的信任关系。我们为设计提出了一些启示，讨论了从业者在实践中面临的解释性挑战与文献提出的方法之间的差距，并强调了可能更好地满足实际需求的研究方向。

模型可解释性中的人因因素：行业实践，挑战和需求