Video-grounded Dialogue (VGD) aims to decode an answer sentence to a question
regarding a given video and dialogue context. Despite the recent success of
multi-modal reasoning to generate answer sentences, existing dialogue systems
still suffer from a text hallucination problem, which denotes indiscriminate
text-copying from input texts without an understanding of the question. This is
due to learning spurious correlations from the fact that answer sentences in
the dataset usually include the words of input texts, thus the VGD system
excessively relies on copying words from input texts by hoping those words to
overlap with ground-truth texts. Hence, we design Text Hallucination Mitigating
(THAM) framework, which incorporates Text Hallucination Regularization (THR)
loss derived from the proposed information-theoretic text hallucination
measurement approach. Applying THAM with current dialogue systems validates the
effectiveness on VGD benchmarks (i.e., AVSD@DSTC7 and AVSD@DSTC8) and shows
enhanced interpretability.

该研究设计了一种文本幻觉缓解框架（THAM），并通过当前的对话系统运用该框架验证了其在基准测试（即 AVSD @ DSTC7 和 AVSD @ DSTC8）上对 Video-grounded 对话的有效性和提高的解释性。

基于信息论的视频对话文本幻觉减少

Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue

With increasing information from social media, there are more and more videos
available. Therefore, the ability to reason on a video is important and
deserves to be discussed. TheDialog System Technology Challenge (DSTC7)
(Yoshino et al. 2018) proposed an Audio Visual Scene-aware Dialog (AVSD) task,
which contains five modalities including video, dialogue history, summary, and
caption, as a scene-aware environment. In this paper, we propose the
entropy-enhanced dynamic memory network (DMN) to effectively model video
modality. The attention-based GRU in the proposed model can improve the model's
ability to comprehend and memorize sequential information. The entropy
mechanism can control the attention distribution higher, so each to-be-answered
question can focus more specifically on a small set of video segments. After
the entropy-enhanced DMN secures the video context, we apply an attention model
that in-corporates summary and caption to generate an accurate answer given the
question about the video. In the official evaluation, our system can achieve
improved performance against the released baseline model for both subjective
and objective evaluation metrics.

本研究提出了熵增强动态内存网络 (DMN) 以有效地对视频模态建模，并应用基于注意力的 GRU 来提高模型理解和记忆序列信息的能力，从而生成对于视频问答问题精确的回答。在官方评估中，我们的系统在主观和客观评估指标上均能实现比已发布的基准模型更好的性能。

基于熵增强的多模态注意力模型用于场景感知对话生成

Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue  Generation

Scene-aware dialog systems will be able to have conversations with users
about the objects and events around them. Progress on such systems can be made
by integrating state-of-the-art technologies from multiple research areas
including end-to-end dialog systems visual dialog, and video description. We
introduce the Audio Visual Scene Aware Dialog (AVSD) challenge and dataset. In
this challenge, which is one track of the 7th Dialog System Technology
Challenges (DSTC7) workshop1, the task is to build a system that generates
responses in a dialog about an input video

该研究论文介绍了一种基于多个研究领域的技术相互整合的新型场景感知对话系统，同时提出了基于音视频的场景感知对话系统 (AVSD) 挑战与数据集，该挑战要求参赛者构建一个能够对输入视频进行对话响应的系统。