Explanation(attention)-guided learning is a method that enhances a model's predictive power by incorporating human understanding during the training phase. While attention-guided learning has shown promising results, it often involves time-consuming and computationally expensive model retraining. To address this issue, we introduce the attention-prompted prediction technique, which enables direct prediction guided by the attention prompt without the need for model retraining. However, this approach presents several challenges, including: 1) How to incorporate the visual attention prompt into the model's decision-making process and leverage it for future predictions even in the absence of a prompt? and 2) How to handle the incomplete information from the visual attention prompt? To tackle these challenges, we propose a novel framework called Visual Attention-Prompted Prediction and Learning, which seamlessly integrates visual attention prompts into the model's decision-making process and adapts to images both with and without attention prompts for prediction. To address the incomplete information of the visual attention prompt, we introduce a perturbation-based attention map modification method. Additionally, we propose an optimization-based mask aggregation method with a new weight learning function for adaptive perturbed annotation aggregation in the attention map modification process. Our overall framework is designed to learn in an attention-prompt guided multi-task manner to enhance future predictions even for samples without attention prompts and trained in an alternating manner for better convergence. Extensive experiments conducted on two datasets demonstrate the effectiveness of our proposed framework in enhancing predictions for samples, both with and without provided prompts.

Visual Attention-Prompted Prediction and Learning是一种新的框架，可以将视觉关注提示无缝地融入模型的决策过程，并适应具有和没有关注提示的图像进行预测。为了解决视觉关注提示的不完整信息问题，我们引入了基于扰动的关注图修改方法，并提出了一种优化的蒙版聚合方法以及新的权重学习函数，以适应关注图修改过程中的自适应扰动标注聚合。我们的整体框架是为了在多任务的关注提示引导下学习，以提高未提供关注提示的样本的未来预测能力，并通过交替训练的方式来获得更好的收敛性。在两个数据集上进行的大量实验证明了我们提出的框架在增强具有和无关注提示的样本的预测能力方面的有效性。

视觉注意引导的预测与学习