Understanding long, real-world videos requires modeling of long-range visual
dependencies. To this end, we explore video-first architectures, building on
the common paradigm of transferring large-scale, image--text models to video
via shallow temporal fusion. However, we expose two limitations to the
approach: (1) decreased spatial capabilities, likely due to poor
video--language alignment in standard video datasets, and (2) higher memory
consumption, bottlenecking the number of frames that can be processed. To
mitigate the memory bottleneck, we systematically analyze the memory/accuracy
trade-off of various efficient methods: factorized attention,
parameter-efficient image-to-video adaptation, input masking, and
multi-resolution patchification. Surprisingly, simply masking large portions of
the video (up to 75%) during contrastive pre-training proves to be one of the
most robust ways to scale encoders to videos up to 4.3 minutes at 1 FPS. Our
simple approach for training long video-to-text models, which scales to 1B
parameters, does not add new architectural complexity and is able to outperform
the popular paradigm of using much larger LLMs as an information aggregator
over segment-based information on benchmarks with long-range temporal
dependencies (YouCook2, EgoSchema).

我们研究了长时间、真实世界的视频，发现了视频与文本模型的两个限制：空间能力不足和内存消耗过高。通过分析不同的高效方法进行内存 / 准确度权衡，我们发现简单地遮盖视频的大部分内容是扩展编码器的一种最有效方式。我们提出的简单方法，能够训练具有 10 亿参数的长视频到文本模型，并在长时间依赖性基准测试中胜过更大的信息聚合模型。

对比预训练视频优先编码器的简单配方：超过 16 帧

A Simple Recipe for Contrastively Pre-training Video-First Encoders  Beyond 16 Frames

Efficient Neural Architecture Search methods based on weight sharing have
shown good promise in democratizing Neural Architecture Search for computer
vision models. There is, however, an ongoing debate whether these efficient
methods are significantly better than random search. Here we perform a thorough
comparison between efficient and random search methods on a family of
progressively larger and more challenging search spaces for image
classification and detection on ImageNet and COCO. While the efficacies of both
methods are problem-dependent, our experiments demonstrate that there are
large, realistic tasks where efficient search methods can provide substantial
gains over random search. In addition, we propose and evaluate techniques which
improve the quality of searched architectures and reduce the need for manual
hyper-parameter tuning.
Source code and experiment data are available at
this https URL

本文比较了基于权重共享的高效神经结构搜索算法和随机搜索方法在不同问题领域下的性能表现；同时，该文提出并评估了一些技术手段可用于改进搜索到的结构的质量，进一步降低人工超参数调参的需求。