We present Long Short-term TRansformer (LSTR), a temporal modeling algorithm
for online action detection, which employs a long- and short-term memory
mechanism to model prolonged sequence data. It consists of an LSTR encoder that
dynamically leverages coarse-scale historical information from an extended
temporal window (e.g., 2048 frames spanning of up to 8 minutes), together with
an LSTR decoder that focuses on a short time window (e.g., 32 frames spanning 8
seconds) to model the fine-scale characteristics of the data. Compared to prior
work, LSTR provides an effective and efficient method to model long videos with
fewer heuristics, which is validated by extensive empirical analysis. LSTR
achieves state-of-the-art performance on three standard online action detection
benchmarks, THUMOS'14, TVSeries, and HACS Segment. Code has been made available
at: this https URL

提出了一种长短时 Transformer（LSTR）算法，用于在线动作检测，它采用了长短时记忆机制来建模长序列数据，其中编码器动态地利用拓展的时间窗口的历史信息，解码器专注于一个短时间窗口来建模数据的细节特征。与先前的工作相比，LSTR 提供了一种有效且高效的建模长视频的方法，其在三种标准的在线动作检测基准测试（THUMOS'14、TVSeries 和 HACS Segment）上实现了最先进的性能。

在线动作检测的长短期 Transformer

Long Short-Term Transformer for Online Action Detection

This paper investigates how to realize better and more efficient embedding
learning to tackle the semi-supervised video object segmentation under
challenging multi-object scenarios. The state-of-the-art methods learn to
decode features with a single positive object and thus have to match and
segment each target separately under multi-object scenarios, consuming multiple
times computing resources. To solve the problem, we propose an Associating
Objects with Transformers (AOT) approach to match and decode multiple objects
uniformly. In detail, AOT employs an identification mechanism to associate
multiple targets into the same high-dimensional embedding space. Thus, we can
simultaneously process multiple objects' matching and segmentation decoding as
efficiently as processing a single object. For sufficiently modeling
multi-object association, a Long Short-Term Transformer is designed for
constructing hierarchical matching and propagation. We conduct extensive
experiments on both multi-object and single-object benchmarks to examine AOT
variant networks with different complexities. Particularly, our R50-AOT-L
outperforms all the state-of-the-art competitors on three popular benchmarks,
i.e., YouTube-VOS (84.1% J&F), DAVIS 2017 (84.9%), and DAVIS 2016 (91.1%),
while keeping more than $3\times$ faster multi-object run-time. Meanwhile, our
AOT-T can maintain real-time multi-object speed on the above benchmarks. Based
on AOT, we ranked 1st in the 3rd Large-scale VOS Challenge.

本研究提出了基于 AOT 的嵌入式学习方法，通过将多个对象相互关联在同一高维度嵌入空间中，实现了同时处理多个对象的匹配和分割解码，具有更高的效率，并且在多个基准测试中均优于已有的 State-of-the-art 算法，是第三次大规模 VOS 挑战的第一名。