In recent years, deep saliency models have made significant progress in
predicting human visual attention. However, the mechanisms behind their success
remain largely unexplained due to the opaque nature of deep neural networks. In
this paper, we present a novel analytic framework that sheds light on the
implicit features learned by saliency models and provides principled
interpretation and quantification of their contributions to saliency
prediction. Our approach decomposes these implicit features into interpretable
bases that are explicitly aligned with semantic attributes and reformulates
saliency prediction as a weighted combination of probability maps connecting
the bases and saliency. By applying our framework, we conduct extensive
analyses from various perspectives, including the positive and negative weights
of semantics, the impact of training data and architectural designs, the
progressive influences of fine-tuning, and common failure patterns of
state-of-the-art deep saliency models. Additionally, we demonstrate the
effectiveness of our framework by exploring visual attention characteristics in
various application scenarios, such as the atypical attention of people with
autism spectrum disorder, attention to emotion-eliciting stimuli, and attention
evolution over time. Our code is publicly available at
https://github.com/szzexpoi/saliency_analysis.

通过分解隐含特征为可解释的基底，提出了一种新的分析框架，用于研究深度显著性模型成功背后的隐含特征、显著性预测的原理以及其对不同应用场景中的视觉注意力的影响。

深度显著性模型对视觉注意力的学习

What Do Deep Saliency Models Learn about Visual Attention?

Generative Adversarial Networks (GANs) have achieved impressive results for
many real-world applications. As an active research topic, many GAN variants
have emerged with improvements in sample quality and training stability.
However, visualization and understanding of GANs is largely missing. How does a
GAN represent our visual world internally? What causes the artifacts in GAN
results? How do architectural choices affect GAN learning? Answering such
questions could enable us to develop new insights and better models. In this
work, we present an analytic framework to visualize and understand GANs at the
unit-, object-, and scene-level. We first identify a group of interpretable
units that are closely related to concepts with a segmentation-based network
dissection method. We quantify the causal effect of interpretable units by
measuring the ability of interventions to control objects in the output.
Finally, we examine the contextual relationship between these units and their
surrounding by inserting the discovered concepts into new images. We show
several practical applications enabled by our framework, from comparing
internal representations across different layers, models, and datasets, to
improving GANs by locating and removing artifact-causing units, to
interactively manipulating objects in the scene. We will open source our
interactive tools to help researchers and practitioners better understand their
models.

研究了生成对抗网络（GANs）在内部如何表示我们的视觉世界，探讨了 GANs 结果中众所周知的人工痕迹是如何引起的，并展示了一个可视化和理解 GANs 的分析框架，旨在使研究者和从业者更好地理解其模型并进行实际应用，例如跨不同层，模型和数据集比较内部表示，定位和移除人工痕迹引起的单元，以及交互式操纵场景中的对象。