In this paper, we present a novel methodology we call MDS-ViTNet (Multi
Decoder Saliency by Vision Transformer Network) for enhancing visual saliency
prediction or eye-tracking. This approach holds significant potential for
diverse fields, including marketing, medicine, robotics, and retail. We propose
a network architecture that leverages the Vision Transformer, moving beyond the
conventional ImageNet backbone. The framework adopts an encoder-decoder
structure, with the encoder utilizing a Swin transformer to efficiently embed
most important features. This process involves a Transfer Learning method,
wherein layers from the Vision Transformer are converted by the Encoder
Transformer and seamlessly integrated into a CNN Decoder. This methodology
ensures minimal information loss from the original input image. The decoder
employs a multi-decoding technique, utilizing dual decoders to generate two
distinct attention maps. These maps are subsequently combined into a singular
output via an additional CNN model. Our trained model MDS-ViTNet achieves
state-of-the-art results across several benchmarks. Committed to fostering
further collaboration, we intend to make our code, models, and datasets
accessible to the public.

本文介绍了一种名为 MDS-ViTNet（多解码视觉变换网络）的新方法，用于增强视觉显著性预测和眼动跟踪。该方法在多个领域具有重要潜力，包括市场营销、医学、机器人技术和零售业。我们提出了一种利用 Vision Transformer 的网络架构，超越传统的 ImageNet 骨干。该框架采用编码器 - 解码器结构，其中编码器利用 Swin 变换器有效地嵌入最重要的特征。通过转换器将 Vision Transformer 的层转换并无缝集成到 CNN 解码器中，从而实现了该过程。编码器 - 解码器方法能够最小化原始输入图像的信息损失。解码器采用多解码技术，利用双解码器生成两个不同的关注度图。通过附加的 CNN 模型将这些图合并为单一输出。我们训练的模型 MDS-ViTNet 在多个基准测试中实现了最新的结果。为了促进进一步的合作，我们计划将我们的代码、模型和数据集开放给公众使用。

MDS-ViTNet: 基于视觉 Transformer 改进注视预测的方法

MDS-ViTNet: Improving saliency prediction for Eye-Tracking with Vision  Transformer

Recently, with the advent of deep convolutional neural networks (DCNN), the
improvements in visual saliency prediction research are impressive. One
possible direction to approach the next improvement is to fully characterize
the multi-scale saliency-influential factors with a computationally-friendly
module in DCNN architectures. In this work, we proposed an end-to-end dilated
inception network (DINet) for visual saliency prediction. It captures
multi-scale contextual features effectively with very limited extra parameters.
Instead of utilizing parallel standard convolutions with different kernel sizes
as the existing inception module, our proposed dilated inception module (DIM)
uses parallel dilated convolutions with different dilation rates which can
significantly reduce the computation load while enriching the diversity of
receptive fields in feature maps. Moreover, the performance of our saliency
model is further improved by using a set of linear normalization-based
probability distribution distance metrics as loss functions. As such, we can
formulate saliency prediction as a probability distribution prediction task for
global saliency inference instead of a typical pixel-wise regression problem.
Experimental results on several challenging saliency benchmark datasets
demonstrate that our DINet with proposed loss functions can achieve
state-of-the-art performance with shorter inference time.

本文提出了一种基于 DCNN 网络的端到端扩张 Inception 网络（DINet）用于实现视觉显著性预测，并使用一系列线性归一化的概率分布距离度量作为损失函数来推断全局显著性信息，从而将显著性预测转化为概率分布任务。实验结果表明，该方法可以在短时间内实现最先进的性能。

基于扩张 Inception 网络的视觉显著性预测

A Dilated Inception Network for Visual Saliency Prediction

This paper revisits visual saliency prediction by evaluating the recent
advancements in this field such as crowd-sourced mouse tracking-based databases
and contextual annotations. We pursue a critical and quantitative approach
towards some of the new challenges including the quality of mouse tracking
versus eye tracking for model training and evaluation. We extend quantitative
evaluation of models in order to incorporate contextual information by
proposing an evaluation methodology that allows accounting for contextual
factors such as text, faces, and object attributes. The proposed contextual
evaluation scheme facilitates detailed analysis of models and helps identify
their pros and cons. Through several experiments, we find that (1) mouse
tracking data has lower inter-participant visual congruency and higher
dispersion, compared to the eye tracking data, (2) mouse tracking data does not
totally agree with eye tracking in general and in terms of different contextual
regions in specific, and (3) mouse tracking data leads to acceptable results in
training current existing models, and (4) mouse tracking data is less reliable
for model selection and evaluation. The contextual evaluation also reveals
that, among the studied models, there is no single model that performs best on
all the tested annotations.

通过对视觉显著性预测的量化评估，本文探讨最新挑战，包括对模型训练和评估的鼠标跟踪数据和眼动跟踪数据质量的比较，提出了一种考虑文本、面部和对象属性等背景信息的评估方法，并发现在所有测试的注释中没有单一的模型表现最佳。