In contrast to conventional visual question answering, video-grounded dialog
necessitates a profound understanding of both dialog history and video content
for accurate response generation. Despite commendable strides made by existing
methodologies, they often grapple with the challenges of incrementally
understanding intricate dialog histories and assimilating video information. In
response to this gap, we present an iterative tracking and reasoning strategy
that amalgamates a textual encoder, a visual encoder, and a generator. At its
core, our textual encoder is fortified with a path tracking and aggregation
mechanism, adept at gleaning nuances from dialog history that are pivotal to
deciphering the posed questions. Concurrently, our visual encoder harnesses an
iterative reasoning network, meticulously crafted to distill and emphasize
critical visual markers from videos, enhancing the depth of visual
comprehension. Culminating this enriched information, we employ the pre-trained
GPT-2 model as our response generator, stitching together coherent and
contextually apt answers. Our empirical assessments, conducted on two renowned
datasets, testify to the prowess and adaptability of our proposed design.

对比传统的视觉问答，基于视频的对话需要深入理解对话历史和视频内容以实现准确的响应生成。为了解决现有方法在逐步理解复杂对话历史和融入视频信息方面所面临的挑战，我们提出了一种迭代的跟踪与推理策略，将文本编码器、视觉编码器和生成器相结合。在核心部分，我们的文本编码器具有路径追踪和聚合机制，能够从对话历史中提取对解读提问至关重要的细微差别。同时，我们的视觉编码器采用迭代推理网络，精心设计以从视频中提取和强调关键的视觉标记，增强视觉理解的深度。通过使用预训练的 GPT-2 模型作为响应生成器，将这些丰富的信息整合在一起，生成连贯和与上下文相关的答案。我们在两个有名的数据集上进行的实证评估证实了我们提出设计的实力和适应性。