Effective communication between humans and intelligent agents has promising
applications for solving complex problems. One such approach is visual
dialogue, which leverages multimodal context to assist humans. However,
real-world scenarios occasionally involve human mistakes, which can cause
intelligent agents to fail. While most prior research assumes perfect answers
from human interlocutors, we focus on a setting where the agent points out
unintentional mistakes for the interlocutor to review, better reflecting
real-world situations. In this paper, we show that human answer mistakes depend
on question type and QA turn in the visual dialogue by analyzing a previously
unused data collection of human mistakes. We demonstrate the effectiveness of
those factors for the model's accuracy in a pointing-human-mistake task through
experiments using a simple MLP model and a Visual Language Model.

通过分析以前未使用的人类错误数据集，展示了人类回答错误与问题类型和视觉对话中的问答轮次的相关性，并通过使用简单的 MLP 模型和视觉语言模型的实验，证明了这些因素对于模型准确性在指向人类错误任务中的有效性。

在目标导向的视觉对话中指出人类答案错误

Pointing out Human Answer Mistakes in a Goal-Oriented Visual Dialogue

The Conversational Question Answering (CoQA) task involves answering a
sequence of inter-related conversational questions about a contextual
paragraph. Although existing approaches employ human-written ground-truth
answers for answering conversational questions at test time, in a realistic
scenario, the CoQA model will not have any access to ground-truth answers for
the previous questions, compelling the model to rely upon its own previously
predicted answers for answering the subsequent questions. In this paper, we
find that compounding errors occur when using previously predicted answers at
test time, significantly lowering the performance of CoQA systems. To solve
this problem, we propose a sampling strategy that dynamically selects between
target answers and model predictions during training, thereby closely
simulating the situation at test time. Further, we analyse the severity of this
phenomena as a function of the question type, conversation length and domain
type.

本文针对谈话式问答（CoQA）任务中可能出现的由于模型先前的预测答案导致的错误累积问题，提出了一种采样策略，旨在在训练期间动态地在目标答案和模型预测之间进行选择，并分析了此现象的严重程度与问题类型，对话长度和域类型的关系。