We use multilayer Long Short Term Memory (LSTM) networks to learn
representations of video sequences. Our model uses an encoder LSTM to map an
input sequence into a fixed length representation. This representation is
decoded using single or multiple decoder LSTMs to perform different tasks, such
as reconstructing the input sequence, or predicting the future sequence. We
experiment with two kinds of input sequences - patches of image pixels and
high-level representations ("percepts") of video frames extracted using a
pretrained convolutional net. We explore different design choices such as
whether the decoder LSTMs should condition on the generated output. We analyze
the outputs of the model qualitatively to see how well the model can
extrapolate the learned video representation into the future and into the past.
We try to visualize and interpret the learned features. We stress test the
model by running it on longer time scales and on out-of-domain data. We further
evaluate the representations by finetuning them for a supervised learning
problem - human action recognition on the UCF-101 and HMDB-51 datasets. We show
that the representations help improve classification accuracy, especially when
there are only a few training examples. Even models pretrained on unrelated
datasets (300 hours of YouTube videos) can help action recognition performance.

使用多层 LSTM 网络来学习视频序列的表示，采用编码器 LSTM 将输入序列映射成固定长度的表示，通过单个或多个解码器 LSTM 解码以执行不同的任务，如重构输入序列或预测未来序列，实验说明预先训练好的模型在人类动作识别等任务上的有效性。