When watching a video, humans can naturally extract human actions from the
surrounding scene context, even when action-scene combinations are unusual.
However, unlike humans, video action recognition models often learn
scene-biased action representations from the spurious correlation in training
data, leading to poor performance in out-of-context scenarios. While
scene-debiased models achieve improved performance in out-of-context scenarios,
they often overlook valuable scene information in the data. Addressing this
challenge, we propose Disentangled VIdeo representations of Action and Scene
(DEVIAS), which aims to achieve holistic video understanding. Disentangled
action and scene representations with our method could provide flexibility to
adjust the emphasis on action or scene information depending on downstream task
and dataset characteristics. Disentangled action and scene representations
could be beneficial for both in-context and out-of-context video understanding.
To this end, we employ slot attention to learn disentangled action and scene
representations with a single model, along with auxiliary tasks that further
guide slot attention. We validate the proposed method on both in-context
datasets: UCF-101 and Kinetics-400, and out-of-context datasets: SCUBA and HAT.
Our proposed method shows favorable performance across different datasets
compared to the baselines, demonstrating its effectiveness in diverse video
understanding scenarios.

通过 Disentangled VIdeo representations of Action and Scene (DEVIAS) 提出了一种细分行动和场景表示的方法，利用 slot attention 和辅助任务来有效提高视频理解性能。