In this paper, we tackle the problem of egocentric action anticipation, i.e.,
predicting what actions the camera wearer will perform in the near future and
which objects they will interact with. Specifically, we contribute
Rolling-Unrolling LSTM, a learning architecture to anticipate actions from
egocentric videos. The method is based on three components: 1) an architecture
comprised of two LSTMs to model the sub-tasks of summarizing the past and
inferring the future, 2) a Sequence Completion Pre-Training technique which
encourages the LSTMs to focus on the different sub-tasks, and 3) a Modality
ATTention (MATT) mechanism to efficiently fuse multi-modal predictions
performed by processing RGB frames, optical flow fields and object-based
features. The proposed approach is validated on EPIC-Kitchens, EGTEA Gaze+ and
ActivityNet. The experiments show that the proposed architecture is
state-of-the-art in the domain of egocentric videos, achieving top performances
in the 2019 EPIC-Kitchens egocentric action anticipation challenge. The
approach also achieves competitive performance on ActivityNet with respect to
methods not based on unsupervised pre-training and generalizes to the tasks of
early action recognition and action recognition. To encourage research on this
challenging topic, we made our code, trained models, and pre-extracted features
available at our web page: this http URL

本文提出了一种基于 Rolling-Unrolling LSTM、序列完成预训练技术和 Modality ATTention 机制的方法来预测自我中心视频中未来的行动和交互对象，并在 EPIC-Kitchens、EGTEA Gaze + 和 ActivityNet 数据集上验证了其有效性。

基于第一人称视频的动作预测的滚动展开 LSTM

Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video

Egocentric action anticipation consists in understanding which objects the
camera wearer will interact with in the near future and which actions they will
perform. We tackle the problem proposing an architecture able to anticipate
actions at multiple temporal scales using two LSTMs to 1) summarize the past,
and 2) formulate predictions about the future. The input video is processed
considering three complimentary modalities: appearance (RGB), motion (optical
flow) and objects (object-based features). Modality-specific predictions are
fused using a novel Modality ATTention (MATT) mechanism which learns to weigh
modalities in an adaptive fashion. Extensive evaluations on two large-scale
benchmark datasets show that our method outperforms prior art by up to +7% on
the challenging EPIC-Kitchens dataset including more than 2500 actions, and
generalizes to EGTEA Gaze+. Our approach is also shown to generalize to the
tasks of early action recognition and action recognition. Our method is ranked
first in the public leaderboard of the EPIC-Kitchens egocentric action
anticipation challenge 2019. Please see our web pages for code and examples:
this http URL - this https URL

本研究提出了一种基于 LSTMs 和 Modality ATTention (MATT) 机制的行为预测框架，能够对计算机视觉领域的尺度序列问题进行有效解决，同时在 EPIC-Kitchens 等数据集上表现出色。

用 Rolling-Unrolling LSTM 和模态注意力预测以自我为中心的行为

What Would You Expect? Anticipating Egocentric Actions with  Rolling-Unrolling LSTMs and Modality Attention

We introduce a new task called Multimodal Named Entity Recognition (MNER) for
noisy user-generated data such as tweets or Snapchat captions, which comprise
short text with accompanying images. These social media posts often come in
inconsistent or incomplete syntax and lexical notations with very limited
surrounding textual contexts, bringing significant challenges for NER. To this
end, we create a new dataset for MNER called SnapCaptions (Snapchat
image-caption pairs submitted to public and crowd-sourced stories with fully
annotated named entities). We then build upon the state-of-the-art Bi-LSTM
word/character based NER models with 1) a deep image network which incorporates
relevant visual context to augment textual information, and 2) a generic
modality-attention module which learns to attenuate irrelevant modalities while
amplifying the most informative ones to extract contexts from, adaptive to each
sample and token. The proposed MNER model with modality attention significantly
outperforms the state-of-the-art text-only NER models by successfully
leveraging provided visual contexts, opening up potential applications of MNER
on myriads of social media platforms.

本研究提出 Multimodal Named Entity Recognition (MNER) 任务，针对用户生成的显式短文本和图像（如 Snapchat 详情）创建了一个新的数据集 SnapCaptions，并利用最先进的 Bi-LSTM 基于词语 / 字符的 NER 模型和深度图像网络结合通用模态注意力模块实现 MNER 任务，成功的通过利用提供的视觉上下文背景比文本 NER 模型具有更佳的性能表现。