Video anomaly detection (VAD) is an important but challenging task in
computer vision. The main challenge rises due to the rarity of training samples
to model all anomaly cases. Hence, semi-supervised anomaly detection methods
have gotten more attention, since they focus on modeling normals and they
detect anomalies by measuring the deviations from normal patterns. Despite
impressive advances of these methods in modeling normal motion and appearance,
long-term motion modeling has not been effectively explored so far. Inspired by
the abilities of the future frame prediction proxy-task, we introduce the task
of future video prediction from a single frame, as a novel proxy-task for video
anomaly detection. This proxy-task alleviates the challenges of previous
methods in learning longer motion patterns. Moreover, we replace the initial
and future raw frames with their corresponding semantic segmentation map, which
not only makes the method aware of object class but also makes the prediction
task less complex for the model. Extensive experiments on the benchmark
datasets (ShanghaiTech, UCSD-Ped1, and UCSD-Ped2) show the effectiveness of the
method and the superiority of its performance compared to SOTA prediction-based
VAD methods.

通过将初始和未来的原始帧替换为相应的语义分割地图，我们引入了从单帧进行未来视频预测的代理任务，以克服以往方法在学习更长动态模式方面的挑战，并展示了该方法在基准数据集上的有效性和优越性。

单帧未来视频预测用于视频异常检测

Future Video Prediction from a Single Frame for Video Anomaly Detection

We study the problem of video-to-video synthesis, whose goal is to learn a
mapping function from an input source video (e.g., a sequence of semantic
segmentation masks) to an output photorealistic video that precisely depicts
the content of the source video. While its image counterpart, the
image-to-image synthesis problem, is a popular topic, the video-to-video
synthesis problem is less explored in the literature. Without understanding
temporal dynamics, directly applying existing image synthesis approaches to an
input video often results in temporally incoherent videos of low visual
quality. In this paper, we propose a novel video-to-video synthesis approach
under the generative adversarial learning framework. Through carefully-designed
generator and discriminator architectures, coupled with a spatio-temporal
adversarial objective, we achieve high-resolution, photorealistic, temporally
coherent video results on a diverse set of input formats including segmentation
masks, sketches, and poses. Experiments on multiple benchmarks show the
advantage of our method compared to strong baselines. In particular, our model
is capable of synthesizing 2K resolution videos of street scenes up to 30
seconds long, which significantly advances the state-of-the-art of video
synthesis. Finally, we apply our approach to future video prediction,
outperforming several state-of-the-art competing systems.

本文提出了一种基于生成对抗学习框架、伴随有空间–时间对抗性目标的视频生成方法，可高效地从语义分割掩膜、素描和姿势等多种输入格式生成 2K 分辨率、30 秒长的逼真视觉效果的视频，此方法成功优化了视频生成技术的最新发展状态，并被应用于未来视频预测。

视频到视频合成

Video-to-Video Synthesis

Future video prediction is an ill-posed Computer Vision problem that recently
received much attention. Its main challenges are the high variability in video
content, the propagation of errors through time, and the non-specificity of the
future frames: given a sequence of past frames there is a continuous
distribution of possible futures. This work introduces bijective Gated
Recurrent Units, a double mapping between the input and output of a GRU layer.
This allows for recurrent auto-encoders with state sharing between encoder and
decoder, stratifying the sequence representation and helping to prevent
capacity problems. We show how with this topology only the encoder or decoder
needs to be applied for input encoding and prediction, respectively. This
reduces the computational cost and avoids re-encoding the predictions when
generating a sequence of frames, mitigating the propagation of errors.
Furthermore, it is possible to remove layers from an already trained model,
giving an insight to the role performed by each layer and making the model more
explainable. We evaluate our approach on three video datasets, outperforming
state of the art prediction results on MMNIST and UCF101, and obtaining
competitive results on KTH with 2 and 3 times less memory usage and
computational cost than the best scored approach.

本研究介绍了双射门控循环单元，为计算机视觉中未来视频预测问题提供了一种有效的方法，能够使编码 / 解码器之间的状态共享并减少计算成本，达到了与最优方法相当的性能。