We introduce PlausiVL, a large video-language model for anticipating action
sequences that are plausible in the real-world. While significant efforts have
been made towards anticipating future actions, prior approaches do not take
into account the aspect of plausibility in an action sequence. To address this
limitation, we explore the generative capability of a large video-language
model in our work and further, develop the understanding of plausibility in an
action sequence by introducing two objective functions, a counterfactual-based
plausible action sequence learning loss and a long-horizon action repetition
loss. We utilize temporal logical constraints as well as verb-noun action pair
logical constraints to create implausible/counterfactual action sequences and
use them to train the model with plausible action sequence learning loss. This
loss helps the model to differentiate between plausible and not plausible
action sequences and also helps the model to learn implicit temporal cues
crucial for the task of action anticipation. The long-horizon action repetition
loss puts a higher penalty on the actions that are more prone to repetition
over a longer temporal window. With this penalization, the model is able to
generate diverse, plausible action sequences. We evaluate our approach on two
large-scale datasets, Ego4D and EPIC-Kitchens-100, and show improvements on the
task of action anticipation.

通过引入 PlausiVL，我们开发了一个大规模视频语言模型，提供了在现实世界中可能发生的行动序列的能力，并引入了两个客观函数，即基于反事实的可行动作序列学习损失和长时程行动重复损失，用以探讨行动序列的可行性。我们通过利用时态逻辑约束和动词 - 名词动作对逻辑约束来创建不可信 / 反事实行动序列，并使用这些序列以合理动作序列学习损失训练模型，以便模型区分可行和不可行的行动序列，并学习隐含的针对行动预测任务的时间暗示。长时程行动重复损失对于在更长的时间窗口内容易重复的动作施加更高的惩罚，增强模型生成多样且可行的行动序列能力。我们在两个大规模数据集（Ego4D 和 EPIC-Kitchens-100）上评估了我们的方法，并在行动预测任务中取得了显著改善。