We introduce LEAP (illustrated in Figure 1), a novel method for generating
video-grounded action programs through use of a Large Language Model (LLM).
These action programs represent the motoric, perceptual, and structural aspects
of action, and consist of sub-actions, pre- and post-conditions, and control
flows. LEAP's action programs are centered on egocentric video and employ
recent developments in LLMs both as a source for program knowledge and as an
aggregator and assessor of multimodal video information. We apply LEAP over a
majority (87\%) of the training set of the EPIC Kitchens dataset, and release
the resulting action programs as a publicly available dataset here
(this https URL).
We employ LEAP as a secondary source of supervision, using its action programs
in a loss term applied to action recognition and anticipation networks. We
demonstrate sizable improvements in performance in both tasks due to training
with the LEAP dataset. Our method achieves 1st place on the EPIC Kitchens
Action Recognition leaderboard as of November 17 among the networks restricted
to RGB-input (see Supplementary Materials).

LEAP 是一种通过使用大型语言模型（Large Language Model）生成视频相关行动程序的新方法，这些行动程序代表行动的动作、感知和结构方面，并包括子动作、前后条件和控制流。通过在 EPIC Kitchens 数据集的大部分训练集上应用 LEAP，并发布生成的行动程序，我们将 LEAP 作为辅助监督源，通过应用其行动程序对行动识别和预测网络施加的损失函数，我们在这两个任务上实现了显著的性能提升。截至 11 月 17 日，我们的方法在 EPIC Kitchens 行动识别排行榜上排名第一。