We present the Object Language Video Transformer (OLViT) - a novel model for video dialog operating over a multi-modal attention-based dialog state tracker. Existing video dialog models struggle with questions requiring both spatial and temporal localization within videos, long-term temporal reasoning, and accurate object tracking across multiple dialog turns. OLViT addresses these challenges by maintaining a global dialog state based on the output of an Object State Tracker (OST) and a Language State Tracker (LST): while the OST attends to the most important objects within the video, the LST keeps track of the most important linguistic co-references to previous dialog turns. In stark contrast to previous works, our approach is generic by nature and is therefore capable of learning continuous multi-modal dialog state representations of the most relevant objects and rounds. As a result, they can be seamlessly integrated into Large Language Models (LLMs) and offer high flexibility in dealing with different datasets and tasks. Evaluations on the challenging DVD (response classification) and SIMMC 2.1 (response generation) datasets show that OLViT achieves new state-of-the-art performance across both datasets.

我们提出了Object Language Video Transformer (OLViT) - 一种新颖的视频对话模型，它在基于多模态注意力的对话状态跟踪器上运行。OLViT通过维护基于对象状态跟踪器 (OST) 和语言状态跟踪器 (LST) 的全局对话状态来解决现有视频对话模型在视频中需要空间和时间定位、长期时间推理和跨多个对话轮次的准确对象跟踪等问题。与之前的工作形成鲜明对比的是，我们的方法和性质是通用的，因此能够学习到最相关的对象和轮次的连续多模态对话状态表示。因此，它们可以无缝地与大型语言模型 (LLM) 集成，并在处理不同数据集和任务时具有高灵活性。在具有挑战性的DVD (响应分类) 和SIMMC 2.1 (响应生成) 数据集上的评估结果显示，OLViT实现了两个数据集上的新的最先进性能。

OLViT: 基于注意力嵌入的视频对话多模态状态追踪