The last years have seen a surge in models predicting the scanpaths of
fixations made by humans when viewing images. However, the field is lacking a
principled comparison of those models with respect to their predictive power.
In the past, models have usually been evaluated based on comparing human
scanpaths to scanpaths generated from the model. Here, instead we evaluate
models based on how well they predict each fixation in a scanpath given the
previous scanpath history. This makes model evaluation closely aligned with the
biological processes thought to underly scanpath generation and allows to apply
established saliency metrics like AUC and NSS in an intuitive and interpretable
way. We evaluate many existing models of scanpath prediction on the datasets
MIT1003, MIT300, CAT2000 train and CAT200 test, for the first time giving a
detailed picture of the current state of the art of human scanpath prediction.
We also show that the discussed method of model benchmarking allows for more
detailed analyses leading to interesting insights about where and when models
fail to predict human behaviour. The MIT/Tuebingen Saliency Benchmark will
implement the evaluation of scanpath models as detailed here, allowing
researchers to score their models on the established benchmark datasets MIT300
and CAT2000.

评估人类扫视路径预测模型的学术论文，引入新的评估指标方法并详细比较了现有模型的性能，提出了以往指标所忽略的问题点。论文最终提出了 MIT/Tuebingen Saliency Benchmark 数据集作为基准，供研究人员参考，评价其模型预测的准确性。

人类扫描路径预测的最新技术

State-of-the-Art in Human Scanpath Prediction

Saliency maps are a popular approach to creating post-hoc explanations of
image classifier outputs. These methods produce estimates of the relevance of
each pixel to the classification output score, which can be displayed as a
saliency map that highlights important pixels. Despite a proliferation of such
methods, little effort has been made to quantify how good these saliency maps
are at capturing the true relevance of the pixels to the classifier output
(i.e. their "fidelity"). We therefore investigate existing metrics for
evaluating the fidelity of saliency methods (i.e. saliency metrics). We find
that there is little consistency in the literature in how such metrics are
calculated, and show that such inconsistencies can have a significant effect on
the measured fidelity. Further, we apply measures of reliability developed in
the psychometric testing literature to assess the consistency of saliency
metrics when applied to individual saliency maps. Our results show that
saliency metrics can be statistically unreliable and inconsistent, indicating
that comparative rankings between saliency methods generated using such metrics
can be untrustworthy.

本研究探索了现有的用于评估显著图质量（即 “保真度”）的指标，并发现当前文献中这些指标的计算方式缺乏一致性，且可重复性差，说明用这些指标生成的显著图质量排名可能不可信。