We introduce STAViS, a spatio-temporal audiovisual saliency network that
combines spatio-temporal visual and auditory information in order to
efficiently address the problem of saliency estimation in videos. Our approach
employs a single network that combines visual saliency and auditory features
and learns to appropriately localize sound sources and to fuse the two
saliencies in order to obtain a final saliency map. The network has been
designed, trained end-to-end, and evaluated on six different databases that
contain audiovisual eye-tracking data of a large variety of videos. We compare
our method against 8 different state-of-the-art visual saliency models.
Evaluation results across databases indicate that our STAViS model outperforms
our visual only variant as well as the other state-of-the-art models in the
majority of cases. Also, the consistently good performance it achieves for all
databases indicates that it is appropriate for estimating saliency
"in-the-wild". The code is available at this https URL

介绍了 STAViS，一种空时音频视觉显著性网络，通过单个网络利用视觉显着性和听觉功能来解决视频中的显著性估计问题，并在多个数据集上性能表现出色，可以用于估算野外环境下的显著性映射。

STAViS: 时空音视频显著性网络

STAViS: Spatio-Temporal AudioVisual Saliency Network

In this work we propose a multi-task spatio-temporal network, called SUSiNet,
that can jointly tackle the spatio-temporal problems of saliency estimation,
action recognition and video summarization. Our approach employs a single
network that is jointly end-to-end trained for all tasks with multiple and
diverse datasets related to the exploring tasks. The proposed network uses a
unified architecture that includes global and task specific layer and produces
multiple output types, i.e., saliency maps or classification labels, by
employing the same video input. Moreover, one additional contribution is that
the proposed network can be deeply supervised through an attention module that
is related to human attention as it is expressed by eye-tracking data. From the
extensive evaluation, on seven different datasets, we have observed that the
multi-task network performs as well as the state-of-the-art single-task methods
(or in some cases better), while it requires less computational budget than
having one independent network per each task.

该研究提出了一种多任务时空网络 ——SUSiNet，可以共同解决显著性估计、动作识别和视频摘要的时空问题。该方法使用一个联合端到端训练的单个网络，使用与探索任务相关的多个数据集。该网络使用统一的体系结构，包括全局和任务特定层，并通过使用相同的视频输入产生多种输出类型。此外，该网络可以通过与人类注意力相关的注意力模块进行深度监督。研究结果表明，该多任务网络的性能与单个任务方法一样好（在某些情况下更好），而且所需的计算预算比每个任务单独使用的网络要少。

SUSiNet: 看见、理解和总结

SUSiNet: See, Understand and Summarize it

Unsupervised video segmentation plays an important role in a wide variety of
applications from object identification to compression. However, to date, fast
motion, motion blur and occlusions pose significant challenges. To address
these challenges for unsupervised video segmentation, we develop a novel
saliency estimation technique as well as a novel neighborhood graph, based on
optical flow and edge cues. Our approach leads to significantly better initial
foreground-background estimates and their robust as well as accurate diffusion
across time. We evaluate our proposed algorithm on the challenging DAVIS,
SegTrack v2 and FBMS-59 datasets. Despite the usage of only a standard edge
detector trained on 200 images, our method achieves state-of-the-art results
outperforming deep learning based methods in the unsupervised setting. We even
demonstrate competitive results comparable to deep learning based methods in
the semi-supervised setting on the DAVIS dataset.

本文提出了基于光流和边缘线索的新型显着性估计技术和新型邻域图，用于解决快速运动，运动模糊和遮挡等视频分割的挑战，在 DAVIS，SegTrack v2 和 FBMS-59 数据集中取得了超越深度学习方法的最新成果。

利用动态显著性引导的时空传播进行无监督视频目标分割

Unsupervised Video Object Segmentation using Motion Saliency-Guided  Spatio-Temporal Propagation

Most saliency estimation methods aim to explicitly model low-level
conspicuity cues such as edges or blobs and may additionally incorporate
top-down cues using face or text detection. Data-driven methods for training
saliency models using eye-fixation data are increasingly popular, particularly
with the introduction of large-scale datasets and deep architectures. However,
current methods in this latter paradigm use loss functions designed for
classification or regression tasks whereas saliency estimation is evaluated on
topographical maps. In this work, we introduce a new saliency map model which
formulates a map as a generalized Bernoulli distribution. We then train a deep
architecture to predict such maps using novel loss functions which pair the
softmax activation function with measures designed to compute distances between
probability distributions. We show in extensive experiments the effectiveness
of such loss functions over standard ones on four public benchmark datasets,
and demonstrate improved performance over state-of-the-art saliency methods.

本研究提出一种基于 Bernoulli 分布的显著性图模型，并使用包括 softmax 在内的新损失函数，通过大规模数据和深度结构进行显著性估计，表现优于现有的显著性方法。