Actions are more than just movements and trajectories: we cook to eat and we
hold a cup to drink from it. A thorough understanding of videos requires going
beyond appearance modeling and necessitates reasoning about the sequence of
activities, as well as the higher-level constructs such as intentions. But how
do we model and reason about these? We propose a fully-connected temporal CRF
model for reasoning over various aspects of activities that includes objects,
actions, and intentions, where the potentials are predicted by a deep network.
End-to-end training of such structured models is a challenging endeavor: For
inference and learning we need to construct mini-batches consisting of whole
videos, leading to mini-batches with only a few videos. This causes
high-correlation between data points leading to breakdown of the backprop
algorithm. To address this challenge, we present an asynchronous variational
inference method that allows efficient end-to-end training. Our method achieves
a classification mAP of 22.4% on the Charades benchmark, outperforming the
state-of-the-art (17.2% mAP), and offers equal gains on the task of temporal
localization.

提出了一种基于完全连接的时间条件随机场模型，用于推理各个活动方面，包括对象、动作和意图，其中潜在函数由深层网络预测；而异步变分推理方法使得高效的端到端训练成为可能，该方法在 Charades 基准测试中达到了 22.4％的分类 mAP，超过了现有技术 (17.2％mAP)。