Video Temporal Grounding (VTG) focuses on accurately identifying event
timestamps within a particular video based on a linguistic query, playing a
vital role in downstream tasks such as video browsing and editing. While Video
Large Language Models (video LLMs) have made significant progress in
understanding video content, they often face challenges in accurately
pinpointing timestamps within videos, which limits their performance on VTG
tasks. Therefore, to improve video LLMs' ability to effectively locate
timestamps, we argue that two critical aspects need to be enhanced. First, it
is essential to have high-quality instructional tuning datasets that encompass
mainstream VTG tasks. Second, directly incorporating timestamp knowledge into
video LLMs is crucial, as it enables models to efficiently comprehend timestamp
information. To address these needs, we first introduce VTG-IT-120K, a
high-quality and comprehensive instruction tuning dataset that covers VTG tasks
such as moment retrieval, dense video captioning, video summarization, and
video highlight detection. Furthermore, we propose a specially designed video
LLM model for VTG tasks, VTG-LLM, which (1) effectively integrates timestamp
knowledge into visual tokens; (2) incorporates absolute-time tokens that
specifically handle timestamp knowledge, thereby avoiding concept shifts; and
(3) introduces a lightweight, high-performance slot-based token compression
method to facilitate the sampling of more video frames. Comprehensive
experiments showcase the superior performance of VTG-LLM in comparison to other
video LLM methods across various VTG tasks. Our code and datasets are available
at https://github.com/gyxxyg/VTG-LLM.

基于 VTG 任务，本研究首先介绍了 VTG-IT-120K 这一高质量的综合指导调整数据集，其涵盖了时刻检索、密集视频字幕生成、视频摘要和视频亮点检测等 VTG 任务；其次，我们提出了一个特别设计的用于 VTG 任务的视频 LLM 模型，VTG-LLM，该模型能够有效地将时间戳知识与视觉标记结合起来，并且引入了一种轻量级、高性能的基于槽位的标记压缩方法，以便更好地采样更多的视频帧。全面的实验证实了 VTG-LLM 在各种 VTG 任务中相较于其他视频 LLM 方法的卓越性能。

VTG-LLM：将时间戳知识整合到视频 LLMs 中以增强视频时间定位

VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced  Video Temporal Grounding

In this paper, we study the effects of incorporating timestamps, such as
document creation dates, into generation systems. Two types of time-aware
prompts are investigated: (1) textual prompts that encode document timestamps
in natural language sentences; and (2) linear prompts that convert timestamps
into continuous vectors. To explore extrapolation to future data points, we
further introduce a new data-to-text generation dataset, TempWikiBio,
containing more than 4 millions of chronologically ordered revisions of
biographical articles from English Wikipedia, each paired with structured
personal profiles. Through data-to-text generation on TempWikiBio, text-to-text
generation on the content transfer dataset, and summarization on XSum, we show
that linear prompts on encoder and textual prompts improve the generation
quality on all datasets. Despite having less performance drop when testing on
data drawn from a later time, linear prompts focus more on non-temporal
information and are less sensitive to the given timestamps, according to human
evaluations and sensitivity analyses. Meanwhile, textual prompts establish the
association between the given timestamps and the output dates, yielding more
factual temporal information in the output.

探讨利用时间戳，如文档创建日期，将时间信息纳入到生成系统中的影响，研究时间感知提示的两种类型，并引入一个新的数据生成数据集 TempWikiBio，通过数据生成、文本传输和摘要三个数据集展示了使用线性提示和文本提示可以提高所有数据集的生成质量，线性提示对非时间信息更加关注，对时间敏感度较低，而文本提示可以产生更符合事实的时间信息。

文本生成的时间感知提示

Time-aware Prompting for Text Generation

Automatic subtitling is the task of automatically translating the speech of
an audiovisual product into short pieces of timed text, in other words,
subtitles and their corresponding timestamps. The generated subtitles need to
conform to multiple space and time requirements (length, reading speed) while
being synchronised with the speech and segmented in a way that facilitates
comprehension. Given its considerable complexity, automatic subtitling has so
far been addressed through a pipeline of elements that deal separately with
transcribing, translating, segmenting into subtitles and predicting timestamps.
In this paper, we propose the first direct automatic subtitling model that
generates target language subtitles and their timestamps from the source speech
in a single solution. Comparisons with state-of-the-art cascaded models trained
with both in- and out-domain data show that our system provides high-quality
subtitles while also being competitive in terms of conformity, with all the
advantages of maintaining a single model.

本论文提出了一种直接自动生成目标语言字幕及相应时间戳的自动字幕模型，与现有的级联模型相比，在提供高质量字幕的同时也具有更高的一致性和维护单一模型的优点。