Visual Question Answering (VQA) is a challenging task that requires
cross-modal understanding and reasoning of visual image and natural language
question. To inspect the association of VQA models to human cognition, we
designed a survey to record human thinking process and analyzed VQA models by
comparing the outputs and attention maps with those of humans. We found that
although the VQA models resemble human cognition in architecture and performs
similarly with human on the recognition-level, they still struggle with
cognitive inferences. The analysis of human thinking procedure serves to direct
future research and introduce more cognitive capacity into modeling features
and architectures.

通过比较视觉问答模型的输出和注意力图与人类的相似之处，研究发现虽然 VQA 模型在结构和识别水平上与人类相似，但在认知推理方面仍然存在挑战。人类思维过程的分析有助于指导未来的研究，并在建模特征和架构中引入更多的认知能力。

视觉问答模型和人类智能的认知比较研究

On the Cognition of Visual Question Answering Models and Human  Intelligence: A Comparative Study

Even to a simple and short news headline, readers react in a multitude of
ways: cognitively (e.g. inferring the writer's intent), emotionally (e.g.
feeling distrust), and behaviorally (e.g. sharing the news with their friends).
Such reactions are instantaneous and yet complex, as they rely on factors that
go beyond interpreting factual content of news. We propose Misinfo Reaction
Frames (MRF), a pragmatic formalism for modeling how readers might react to a
news headline. In contrast to categorical schema, our free-text dimensions
provide a more nuanced way of understanding intent beyond being benign or
malicious. We also introduce a Misinfo Reaction Frames corpus, a crowdsourced
dataset of reactions to over 25k news headlines focusing on global crises: the
Covid-19 pandemic, climate change, and cancer. Empirical results confirm that
it is indeed possible for neural models to predict the prominent patterns of
readers' reactions to previously unseen news headlines. Additionally, our user
study shows that displaying machine-generated MRF implications alongside news
headlines to readers can increase their trust in real news while decreasing
their trust in misinformation. Our work demonstrates the feasibility and
importance of pragmatic inferences on news headlines to help enhance AI-guided
misinformation detection and mitigation.

通过建立自由文本维度的误解反应框架来更准确地了解读者对新闻标题的认知反应，提出 MRF 数据集并使用神经网络学习进行分类预测，证明通过向读者展示机器生成的 MRF 反应可以提高他们对真实新闻的信任。