Predicting human gaze behavior within computer vision is integral for
developing interactive systems that can anticipate user attention, address
fundamental questions in cognitive science, and hold implications for fields
like human-computer interaction (HCI) and augmented/virtual reality (AR/VR)
systems. Despite methodologies introduced for modeling human eye gaze behavior,
applying these models to medical imaging for scanpath prediction remains
unexplored. Our proposed system aims to predict eye gaze sequences from
radiology reports and CXR images, potentially streamlining data collection and
enhancing AI systems using larger datasets. However, predicting human scanpaths
on medical images presents unique challenges due to the diverse nature of
abnormal regions. Our model predicts fixation coordinates and durations
critical for medical scanpath prediction, outperforming existing models in the
computer vision community. Utilizing a two-stage training process and large
publicly available datasets, our approach generates static heatmaps and eye
gaze videos aligned with radiology reports, facilitating comprehensive
analysis. We validate our approach by comparing its performance with
state-of-the-art methods and assessing its generalizability among different
radiologists, introducing novel strategies to model radiologists' search
patterns during CXR image diagnosis. Based on the radiologist's evaluation,
MedGaze can generate human-like gaze sequences with a high focus on relevant
regions over the CXR images. It sometimes also outperforms humans in terms of
redundancy and randomness in the scanpaths.

利用计算机视觉预测医学图像中的人眼注视路径，通过大型数据库生成热度图和眼动视频以协助综合分析，并与最先进的方法相比较其性能和普适性。

多模态学习与放射学认知过程：胸部 X 光扫描路径预测的 MedGaze

Multimodal Learning and Cognitive Processes in Radiology: MedGaze for  Chest X-ray Scanpath Prediction

The prediction of human gaze behavior is important for building
human-computer interactive systems that can anticipate a user's attention.
Computer vision models have been developed to predict the fixations made by
people as they search for target objects. But what about when the image has no
target? Equally important is to know how people search when they cannot find a
target, and when they would stop searching. In this paper, we propose the first
data-driven computational model that addresses the search-termination problem
and predicts the scanpath of search fixations made by people searching for
targets that do not appear in images. We model visual search as an imitation
learning problem and represent the internal knowledge that the viewer acquires
through fixations using a novel state representation that we call Foveated
Feature Maps (FFMs). FFMs integrate a simulated foveated retina into a
pretrained ConvNet that produces an in-network feature pyramid, all with
minimal computational overhead. Our method integrates FFMs as the state
representation in inverse reinforcement learning. Experimentally, we improve
the state of the art in predicting human target-absent search behavior on the
COCO-Search18 dataset

本文提出了一种基于 FFMs 的数据驱动计算模型，用于解决人类搜索终止问题以及预测在图像中搜索不存在目标时人们的扫描路径，通过将 FFMs 集成为状态表示在逆强化学习中将其作为方法，实验结果在 COCO-Search18 数据集上提高了目标缺失搜索行为的预测水平。

无目标人类注意力

Target-absent Human Attention

Being able to predict human gaze behavior has obvious importance for
behavioral vision and for computer vision applications. Most models have mainly
focused on predicting free-viewing behavior using saliency maps, but these
predictions do not generalize to goal-directed behavior, such as when a person
searches for a visual target object. We propose the first inverse reinforcement
learning (IRL) model to learn the internal reward function and policy used by
humans during visual search. The viewer's internal belief states were modeled
as dynamic contextual belief maps of object locations. These maps were learned
by IRL and then used to predict behavioral scanpaths for multiple target
categories. To train and evaluate our IRL model we created COCO-Search18, which
is now the largest dataset of high-quality search fixations in existence.
COCO-Search18 has 10 participants searching for each of 18 target-object
categories in 6202 images, making about 300,000 goal-directed fixations. When
trained and evaluated on COCO-Search18, the IRL model outperformed baseline
models in predicting search fixation scanpaths, both in terms of similarity to
human search behavior and search efficiency. Finally, reward maps recovered by
the IRL model reveal distinctive target-dependent patterns of object
prioritization, which we interpret as a learned object context.

本文提出了第一个反向强化学习模型 (IRL) 模型，使用动态的上下文信念地图来学习人类视觉搜索中的内部奖励函数和策略，并且为了训练和评估我们的 IRL 模型，我们创建了 COCO-Search18 数据集，通过提取学习出的奖励地图，可以了解到目标依赖型背景的学习。