Video language continual learning involves continuously adapting to
information from video and text inputs, enhancing a model's ability to handle
new tasks while retaining prior knowledge. This field is a relatively
under-explored area, and establishing appropriate datasets is crucial for
facilitating communication and research in this field. In this study, we
present the first dedicated benchmark, ViLCo-Bench, designed to evaluate
continual learning models across a range of video-text tasks. The dataset
comprises ten-minute-long videos and corresponding language queries collected
from publicly available datasets. Additionally, we introduce a novel
memory-efficient framework that incorporates self-supervised learning and
mimics long-term and short-term memory effects. This framework addresses
challenges including memory complexity from long video clips, natural language
complexity from open queries, and text-video misalignment. We posit that
ViLCo-Bench, with greater complexity compared to existing continual learning
benchmarks, would serve as a critical tool for exploring the video-language
domain, extending beyond conventional class-incremental tasks, and addressing
complex and limited annotation issues. The curated data, evaluations, and our
novel method are available at this https URL .

本研究提出了第一个专门用于评估在各种视频文本任务中持续学习模型的基准数据集 ViLCo-Bench，并引入了一种新颖的内存高效框架，该框架结合了自监督学习，模拟长期和短期记忆效应，在面对长视频片段的内存复杂性、开放式查询的自然语言复杂性以及文本 - 视频不匹配等挑战时表现出色。