We propose an effective two-stage approach to tackle the problem of
language-based Human-centric Spatio-Temporal Video Grounding (HC-STVG) task. In
the first stage, we propose an Augmented 2D Temporal Adjacent Network
(Augmented 2D-TAN) to temporally ground the target moment corresponding to the
given description. Primarily, we improve the original 2D-TAN from two aspects:
First, a temporal context-aware Bi-LSTM Aggregation Module is developed to
aggregate clip-level representations, replacing the original max-pooling.
Second, we propose to employ Random Concatenation Augmentation (RCA) mechanism
during the training phase. In the second stage, we use pretrained MDETR model
to generate per-frame bounding boxes via language query, and design a set of
hand-crafted rules to select the best matching bounding box outputted by MDETR
for each frame within the grounded moment.

本研究提出了一种有效的两阶段方法，利用增强的 2D-TAN 对文本描述进行时间上的定位，同时使用 MDETR 模型生成每帧包围盒并设计了一组手工规则，筛选出与定位短语最匹配的包围盒。

增强 2D-TAN: 一种用于人类中心的时空视频定位的两阶段方法

Augmented 2D-TAN: A Two-stage Approach for Human-centric Spatio-Temporal  Video Grounding

Multi-modal reasoning systems rely on a pre-trained object detector to
extract regions of interest from the image. However, this crucial module is
typically used as a black box, trained independently of the downstream task and
on a fixed vocabulary of objects and attributes. This makes it challenging for
such systems to capture the long tail of visual concepts expressed in free form
text. In this paper we propose MDETR, an end-to-end modulated detector that
detects objects in an image conditioned on a raw text query, like a caption or
a question. We use a transformer-based architecture to reason jointly over text
and image by fusing the two modalities at an early stage of the model. We
pre-train the network on 1.3M text-image pairs, mined from pre-existing
multi-modal datasets having explicit alignment between phrases in text and
objects in the image. We then fine-tune on several downstream tasks such as
phrase grounding, referring expression comprehension and segmentation,
achieving state-of-the-art results on popular benchmarks. We also investigate
the utility of our model as an object detector on a given label set when
fine-tuned in a few-shot setting. We show that our pre-training approach
provides a way to handle the long tail of object categories which have very few
labelled instances. Our approach can be easily extended for visual question
answering, achieving competitive performance on GQA and CLEVR. The code and
models are available at this https URL

该研究提出了基于 transformer-based 的 MDETR 多模态推理系统，能够从图像中提取物体并进行相关性文本推理，实验表明在对象检测、自然语言问题回答等任务中均获得了最佳表现。