Early action recognition is an important and challenging problem that enables
the recognition of an action from a partially observed video stream where the
activity is potentially unfinished or even not started. In this work, we
propose a novel model that learns a prototypical representation of the full
action for each class and uses it to regularize the architecture and the visual
representations of the partial observations. Our model is very simple in design
and also efficient. We decompose the video into short clips, where a visual
encoder extracts features from each clip independently. Later, a decoder
aggregates together in an online fashion features from all the clips for the
final class prediction. During training, for each partial observation, the
model is jointly trained to both predict the label as well as the action
prototypical representation which acts as a regularizer. We evaluate our method
on multiple challenging real-world datasets and outperform the current
state-of-the-art by a significant margin. For example, on early recognition
observing only the first 10% of each video, our method improves the SOTA by
+2.23 Top-1 accuracy on Something-Something-v2, +3.55 on UCF-101, +3.68 on
SSsub21, and +5.03 on EPIC-Kitchens-55, where prior work used either
multi-modal inputs (e.g. optical-flow) or batched inference. Finally, we also
present exhaustive ablation studies to motivate the design choices we made, as
well as gather insights regarding what our model is learning semantically.

利用部分观察到的视频和原型表示，我们提出了一种早期行为识别模型，通过训练视频的短片段来预测行为并在多个实际数据集上取得了显著的改进。