Video foreground segmentation (VFS) is an important computer vision task
wherein one aims to segment the objects under motion from the background. Most
of the current methods are image-based, i.e., rely only on spatial cues while
ignoring motion cues. Therefore, they tend to overfit the training data and
don't generalize well to out-of-domain (OOD) distribution. To solve the above
problem, prior works exploited several cues such as optical flow, background
subtraction mask, etc. However, having a video data with annotations like
optical flow is a challenging task. In this paper, we utilize the temporal
information and the spatial cues from the video data to improve OOD
performance. However, the challenge lies in how we model the temporal
information given the video data in an interpretable way creates a very
noticeable difference. We therefore devise a strategy that integrates the
temporal context of the video in the development of VFS. Our approach give rise
to deep learning architectures, namely MUSTAN1 and MUSTAN2 and they are based
on the idea of multi-scale temporal context as an attention, i.e., aids our
models to learn better representations that are beneficial for VFS. Further, we
introduce a new video dataset, namely Indoor Surveillance Dataset (ISD) for
VFS. It has multiple annotations on a frame level such as foreground binary
mask, depth map, and instance semantic annotations. Therefore, ISD can benefit
other computer vision tasks. We validate the efficacy of our architectures and
compare the performance with baselines. We demonstrate that proposed methods
significantly outperform the benchmark methods on OOD. In addition, the
performance of MUSTAN2 is significantly improved on certain video categories on
OOD data due to ISD.

视频前景分割是一项重要的计算机视觉任务，本文利用视频数据中的时空信息和空间线索改进了模型的性能，提出了深度学习架构，并引入了 Indoor Surveillance Dataset 以验证其有效性。