The need for more transparency of the decision-making processes in artificial neural networks steadily increases driven by their applications in safety critical and ethically challenging domains such as autonomous driving or medical diagnostics. We address today's lack of transparency of neural networks and shed light on the roles of single neurons and groups of neurons within the network fulfilling a learned task. Inspired by research in the field of neuroscience, we characterize the learned representations by activation patterns and network ablations, revealing functional neuron populations that a) act jointly in response to specific stimuli or b) have similar impact on the network's performance after being ablated. We find that neither a neuron's magnitude or selectivity of activation, nor its impact on network performance are sufficient stand-alone indicators for its importance for the overall task. We argue that such indicators are essential for future advances in transfer learning and modern neuroscience.

本文研究神经网络的透明度缺乏问题，通过对激活模式和网络消融的表征，揭示了在完成学习任务时单个神经元和神经元群体的作用，强调了为未来的转移学习和现代神经科学发展提供必需指标的重要性。

神经网络的内部结构: 基于功能神经元群和网络消融的学习表示特征表征