This survey presents an overview of methods for learning from video (LfV) in
the context of reinforcement learning (RL) and robotics. We focus on methods
capable of scaling to large internet video datasets and, in the process,
extracting foundational knowledge about the world's dynamics and physical human
behaviour. Such methods hold great promise for developing general-purpose
robots.
We open with an overview of fundamental concepts relevant to the
LfV-for-robotics setting. This includes a discussion of the exciting benefits
LfV methods can offer (e.g., improved generalization beyond the available robot
data) and commentary on key LfV challenges (e.g., challenges related to missing
information in video and LfV distribution shifts). Our literature review begins
with an analysis of video foundation model techniques that can extract
knowledge from large, heterogeneous video datasets. Next, we review methods
that specifically leverage video data for robot learning. Here, we categorise
work according to which RL knowledge modality benefits from the use of video
data. We additionally highlight techniques for mitigating LfV challenges,
including reviewing action representations that address the issue of missing
action labels in video.
Finally, we examine LfV datasets and benchmarks, before concluding the survey
by discussing challenges and opportunities in LfV. Here, we advocate for
scalable approaches that can leverage the full range of available data and that
target the key benefits of LfV. Overall, we hope this survey will serve as a
comprehensive reference for the emerging field of LfV, catalysing further
research in the area, and ultimately facilitating progress towards obtaining
general-purpose robots.

该研究综述了在强化学习和机器人领域中学习视频的方法，重点关注能够扩展到大规模互联网视频数据集，并从中提取关于世界动力学和人类行为的基础知识的方法。该综述介绍了学习视频的基本概念、相关挑战以及应对方法，并讨论了学习视频的机器人学习方法和数据集等方面的问题和机会。