In this paper, we tackle the problem of egocentric action anticipation, i.e.,
predicting what actions the camera wearer will perform in the near future and
which objects they will interact with. Specifically, we contribute
Rolling-Unrolling LSTM, a learning architecture to anticipate actions from
egocentric videos. The method is based on three components: 1) an architecture
comprised of two LSTMs to model the sub-tasks of summarizing the past and
inferring the future, 2) a Sequence Completion Pre-Training technique which
encourages the LSTMs to focus on the different sub-tasks, and 3) a Modality
ATTention (MATT) mechanism to efficiently fuse multi-modal predictions
performed by processing RGB frames, optical flow fields and object-based
features. The proposed approach is validated on EPIC-Kitchens, EGTEA Gaze+ and
ActivityNet. The experiments show that the proposed architecture is
state-of-the-art in the domain of egocentric videos, achieving top performances
in the 2019 EPIC-Kitchens egocentric action anticipation challenge. The
approach also achieves competitive performance on ActivityNet with respect to
methods not based on unsupervised pre-training and generalizes to the tasks of
early action recognition and action recognition. To encourage research on this
challenging topic, we made our code, trained models, and pre-extracted features
available at our web page: this http URL

本文提出了一种基于 Rolling-Unrolling LSTM、序列完成预训练技术和 Modality ATTention 机制的方法来预测自我中心视频中未来的行动和交互对象，并在 EPIC-Kitchens、EGTEA Gaze + 和 ActivityNet 数据集上验证了其有效性。