Temporal Sentence Grounding in Videos (TSGV) aims to detect the event
timestamps described by the natural language query from untrimmed videos. This
paper discusses the challenge of achieving efficient computation in TSGV models
while maintaining high performance. Most existing approaches exquisitely design
complex architectures to improve accuracy with extra layers and loss, suffering
from inefficiency and heaviness. Although some works have noticed that, they
only make an issue of feature fusion layers, which can hardly enjoy the
highspeed merit in the whole clunky network. To tackle this problem, we propose
a novel efficient multi-teacher model (EMTM) based on knowledge distillation to
transfer diverse knowledge from both heterogeneous and isomorphic networks.
Specifically, We first unify different outputs of the heterogeneous models into
one single form. Next, a Knowledge Aggregation Unit (KAU) is built to acquire
high-quality integrated soft labels from multiple teachers. After that, the KAU
module leverages the multi-scale video and global query information to
adaptively determine the weights of different teachers. A Shared Encoder
strategy is then proposed to solve the problem that the student shallow layers
hardly benefit from teachers, in which an isomorphic teacher is collaboratively
trained with the student to align their hidden states. Extensive experimental
results on three popular TSGV benchmarks demonstrate that our method is both
effective and efficient without bells and whistles.

本论文提出了一个基于知识蒸馏的新型高效多教师模型（EMTM），旨在在维持高性能的同时，实现在未修剪视频中检测自然语言查询所描述的事件时间戳的挑战。

多教师知识蒸馏下视频中高效的时间句子定位

Efficient Temporal Sentence Grounding in Videos with Multi-Teacher  Knowledge Distillation

Temporal Sentence Grounding in Videos (TSGV), which aims to ground a natural
language sentence in an untrimmed video, has drawn widespread attention over
the past few years. However, recent studies have found that current benchmark
datasets may have obvious moment annotation biases, enabling several simple
baselines even without training to achieve SOTA performance. In this paper, we
take a closer look at existing evaluation protocols, and find both the
prevailing dataset and evaluation metrics are the devils that lead to
untrustworthy benchmarking. Therefore, we propose to re-organize the two
widely-used datasets, making the ground-truth moment distributions different in
the training and test splits, i.e., out-of-distribution (OOD) test. Meanwhile,
we introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic
recall scores to alleviate the inflating evaluation caused by biased datasets.
New benchmarking results indicate that our proposed evaluation protocols can
better monitor the research progress. Furthermore, we propose a novel
causality-based Multi-branch Deconfounding Debiasing (MDD) framework for
unbiased moment prediction. Specifically, we design a multi-branch deconfounder
to eliminate the effects caused by multiple confounders with causal
intervention. In order to help the model better align the semantics between
sentence queries and video moments, we enhance the representations during
feature encoding. Specifically, for textual information, the query is parsed
into several verb-centered phrases to obtain a more fine-grained textual
feature. For visual information, the positional information has been decomposed
from moment features to enhance representations of moments with diverse
locations. Extensive experiments demonstrate that our proposed approach can
achieve competitive results among existing SOTA approaches and outperform the
base model with great gains.

本文提出了一种新的视频文本时间对齐评估协议，包括将常用的数据集重新组织为不同的分布以及引入一种新的评估度量方法。此外，作者还提出了一种基于因果关系的多分支去偏置去混淆框架，帮助模型更好地对齐句子查询和视频片段的语义，实现了更好的性能。

深入探讨视频中去偏置的时间句子定位：数据集、度量和方法

A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach

Temporal Sentence Grounding in Videos (TSGV), i.e., grounding a natural
language sentence which indicates complex human activities in a long and
untrimmed video sequence, has received unprecedented attentions over the last
few years. Although each newly proposed method plausibly can achieve better
performance than previous ones, current TSGV models still tend to capture the
moment annotation biases and fail to take full advantage of multi-modal inputs.
Even more incredibly, several extremely simple baselines without training can
also achieve state-of-the-art performance. In this paper, we take a closer look
at the existing evaluation protocols for TSGV, and find that both the
prevailing dataset splits and evaluation metrics are the devils to cause
unreliable benchmarking. To this end, we propose to re-organize two widely-used
TSGV benchmarks (ActivityNet Captions and Charades-STA). Specifically, we
deliberately make the ground-truth moment distribution different in the
training and test splits, i.e., out-of-distribution (OOD) testing. Meanwhile,
we introduce a new evaluation metric dR@n,IoU@m to calibrate the basic IoU
scores by penalizing on the bias-influenced moment predictions and alleviate
the inflating evaluations caused by the dataset annotation biases such as
overlong ground-truth moments. Under our new evaluation protocol, we conduct
extensive experiments and ablation studies on eight state-of-the-art TSGV
methods. All the results demonstrate that the re-organized dataset splits and
new metric can better monitor the progress in TSGV. Our reorganized datsets are
available at this https URL

本文主要研究 Temporal Sentence Grounding in Videos，在现有的评估协议中，重新组织两个广泛使用的 TSGV 基准及引入新的评估指标 dR @ n，IoU @ m 来校准基本的 IoU 分数，进一步监控 TSGV 的进展。