Language-queried video actor segmentation aims to predict the pixel-level
mask of the actor which performs the actions described by a natural language
query in the target frames. Existing methods adopt 3D CNNs over the video clip
as a general encoder to extract a mixed spatio-temporal feature for the target
frame. Though 3D convolutions are amenable to recognizing which actor is
performing the queried actions, it also inevitably introduces misaligned
spatial information from adjacent frames, which confuses features of the target
frame and yields inaccurate segmentation. Therefore, we propose a collaborative
spatial-temporal encoder-decoder framework which contains a 3D temporal encoder
over the video clip to recognize the queried actions, and a 2D spatial encoder
over the target frame to accurately segment the queried actors. In the decoder,
a Language-Guided Feature Selection (LGFS) module is proposed to flexibly
integrate spatial and temporal features from the two encoders. We also propose
a Cross-Modal Adaptive Modulation (CMAM) module to dynamically recombine
spatial- and temporal-relevant linguistic features for multimodal feature
interaction in each stage of the two encoders. Our method achieves new
state-of-the-art performance on two popular benchmarks with less computational
overhead than previous approaches.

本文提出了一种基于协作的时空编码器 - 解码器框架，其中包含一个三维时间编码器和一个二维空间编码器。在解码器中，提出了一种语言引导的特征选择模块和一种交叉模态适应调制模块，以动态重新组合语言特征和时空相关特征。与以往的方法相比，我们的方法在两个流行的基准测试中实现了新的最先进性能，并且计算开销更小。