Learning from videos is an emerging research area that enables robots to
acquire skills from human demonstrations, such as procedural videos. To do
this, video-language models must be able to obtain structured understandings,
such as the temporal segmentation of a demonstration into sequences of actions
and skills, and to generalize the understandings to novel domains. In pursuit
of this goal, we introduce Spacewalk-18, a benchmark containing two tasks: (1)
step recognition and (2) intra-video retrieval over a dataset of temporally
segmented and labeled tasks in International Space Station spacewalk
recordings. In tandem, the two tasks quantify a model's ability to make use of:
(1) out-of-domain visual information; (2) a high temporal context window; and
(3) multimodal (text + video) domains. This departs from existing benchmarks
for procedural video understanding, which typically deal with short context
lengths and can be solved with a single modality. Spacewalk-18, with its
inherent multimodal and long-form complexity, exposes the high difficulty of
task recognition and segmentation. We find that state-of-the-art methods
perform poorly on our benchmark, demonstrating that the goal of generalizable
procedural video understanding models is far out and underscoring the need to
develop new approaches to these tasks. Data, model, and code will be publicly
released.

视频学习是一个新兴的研究领域，使得机器人能够从人类的示范中获取技能，这需要视频语言模型能够获得结构化的理解，如将示范的时间分段为动作和技能序列，并将这些理解推广到新的领域。本研究引入了 Spacewalk-18 基准，其中包含两个任务：（1）步骤识别和（2）国际空间站太空行走录像中分割和标记任务的内视频检索。这两个任务同时量化了模型利用以下内容的能力：（1）领域外的视觉信息；（2）较长的时间上下文窗口；以及（3）多模态语言（文字 + 视频）领域。与现有的针对程序性视频理解的基准相比，Spacewalk-18 具有固有的多模态和长篇复杂性，揭示了任务识别和分割的高难度。我们发现现有的最先进方法在我们的基准上表现不佳，证明了可推广程序性视频理解模型的目标仍然遥远，并强调了开发新方法以解决这些任务的需要。数据、模型和代码将公开发布。

Spacewalk-18：一项在新领域中进行多模态和长形式过程视频理解的基准测试

Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video  Understanding in Novel Domains

By reducing the curvature of the loss surface in the parameter space,
Sharpness-aware minimization (SAM) yields widespread robustness improvement
under domain transfer. Instead of focusing on parameters, however, this work
considers the transferability of representations as the optimization target for
out-of-domain generalization in a fine-tuning setup. To encourage the retention
of transferable representations, we consider trust region-based fine-tuning
methods, which exploit task-specific skills without forgetting task-agnostic
representations from pre-training. We unify parameter- and representation-space
smoothing approaches by using trust region bounds to inform SAM-style
regularizers on both of these optimization surfaces. We propose Trust Region
Aware Minimization (TRAM), a fine-tuning algorithm that optimizes for flat
minima and smooth, informative representations without forgetting pre-trained
structure. We find that TRAM outperforms both sharpness-aware and trust
region-based optimization methods on cross-domain language modeling and
cross-lingual transfer, where robustness to domain transfer and representation
generality are critical for success. TRAM establishes a new standard in
training generalizable models with minimal additional computation.

通过降低参数空间中损失曲面的曲率，锐度感知最小化（SAM）在域转换下实现了广泛的稳健性改进。本研究不关注参数，而是将表示的可转移性作为优化目标，在微调设置中进行跨域泛化的优化。我们考虑基于信任区域的微调方法，以鼓励保持可转移表示，并通过使用信任区域边界在两个优化表面上通知 SAM 风格的正则化方法来统一参数和表示空间平滑方法。我们提出了一种称为 Trust Region Aware Minimization（TRAM）的微调算法，该算法在不忘记预训练结构的情况下，优化平坦的最小值和平滑的信息表示。我们发现，TRAM 在跨领域语言建模和跨语言转移方面胜过锐度感知和基于信任区域的优化方法，在这些任务中，域转换的健壮性和表示的普适性对于成功至关重要。TRAM 为使用最少额外计算训练可推广模型确立了一个新的标准。