We present the Object Language Video Transformer (OLViT) - a novel model for
video dialog operating over a multi-modal attention-based dialog state tracker.
Existing video dialog models struggle with questions requiring both spatial and
temporal localization within videos, long-term temporal reasoning, and accurate
object tracking across multiple dialog turns. OLViT addresses these challenges
by maintaining a global dialog state based on the output of an Object State
Tracker (OST) and a Language State Tracker (LST): while the OST attends to the
most important objects within the video, the LST keeps track of the most
important linguistic co-references to previous dialog turns. In stark contrast
to previous works, our approach is generic by nature and is therefore capable
of learning continuous multi-modal dialog state representations of the most
relevant objects and rounds. As a result, they can be seamlessly integrated
into Large Language Models (LLMs) and offer high flexibility in dealing with
different datasets and tasks. Evaluations on the challenging DVD (response
classification) and SIMMC 2.1 (response generation) datasets show that OLViT
achieves new state-of-the-art performance across both datasets.

我们提出了 Object Language Video Transformer (OLViT) - 一种新颖的视频对话模型，它在基于多模态注意力的对话状态跟踪器上运行。OLViT 通过维护基于对象状态跟踪器 (OST) 和语言状态跟踪器 (LST) 的全局对话状态来解决现有视频对话模型在视频中需要空间和时间定位、长期时间推理和跨多个对话轮次的准确对象跟踪等问题。与之前的工作形成鲜明对比的是，我们的方法和性质是通用的，因此能够学习到最相关的对象和轮次的连续多模态对话状态表示。因此，它们可以无缝地与大型语言模型 (LLM) 集成，并在处理不同数据集和任务时具有高灵活性。在具有挑战性的 DVD (响应分类) 和 SIMMC 2.1 (响应生成) 数据集上的评估结果显示，OLViT 实现了两个数据集上的新的最先进性能。