Recent works in video prediction have mainly focused on passive forecasting
and low-level action-conditional prediction, which sidesteps the learning of
interaction between agents and objects. We introduce the task of semantic
action-conditional video prediction, which uses semantic action labels to
describe those interactions and can be regarded as an inverse problem of action
recognition. The challenge of this new task primarily lies in how to
effectively inform the model of semantic action information. Inspired by the
idea of Mixture of Experts, we embody each abstract label by a structured
combination of various visual concept learners and propose a novel video
prediction model, Modular Action Concept Network (MAC). Our method is evaluated
on two newly designed synthetic datasets, CLEVR-Building-Blocks and
Sapien-Kitchen, and one real-world dataset called Tower-Creation. Extensive
experiments demonstrate that MAC can correctly condition on given instructions
and generate corresponding future frames without need of bounding boxes. We
further show that the trained model can make out-of-distribution
generalization, be quickly adapted to new object categories and exploit its
learnt features for object detection, showing the progression towards
higher-level cognitive abilities. More visualizations can be found at
this http URL

该研究介绍了使用语义动作标签作为反向问题进行预测的方法，提出了一种名为模块化动作概念网络的视频预测模型，该模型可以生成对应的未来帧且无需边界框，还可以用于物体检测和高层次的认知任务。