Group Activity Recognition (GAR) is a fundamental problem in computer vision,
with diverse applications in sports video analysis, video surveillance, and
social scene understanding. Unlike conventional action recognition, GAR aims to
classify the actions of a group of individuals as a whole, requiring a deep
understanding of their interactions and spatiotemporal relationships. To
address the challenges in GAR, we present REACT (\textbf{R}ecognize
\textbf{E}very \textbf{Act}ion Everywhere All At Once), a novel architecture
inspired by the transformer encoder-decoder model explicitly designed to model
complex contextual relationships within videos, including multi-modality and
spatio-temporal features. Our architecture features a cutting-edge
Vision-Language Encoder block for integrated temporal, spatial, and multi-modal
interaction modeling. This component efficiently encodes spatiotemporal
interactions, even with sparsely sampled frames, and recovers essential local
information. Our Action Decoder Block refines the joint understanding of text
and video data, allowing us to precisely retrieve bounding boxes, enhancing the
link between semantics and visual reality. At the core, our Actor Fusion Block
orchestrates a fusion of actor-specific data and textual features, striking a
balance between specificity and context. Our method outperforms
state-of-the-art GAR approaches in extensive experiments, demonstrating
superior accuracy in recognizing and understanding group activities. Our
architecture's potential extends to diverse real-world applications, offering
empirical evidence of its performance gains. This work significantly advances
the field of group activity recognition, providing a robust framework for
nuanced scene comprehension.

REACT 是一种新颖的架构，受到了 Transformer 编码器 - 解码器模型的启发，旨在模拟视频中复杂的上下文关系，包括多模态和时空特征的交互。该方法在广泛的实验证明，在识别和理解群体活动方面表现出优越的准确性，为细致的场景理解提供了稳健的框架。

REACT：普遍识别每个行为的即刻地点

REACT: Recognize Every Action Everywhere All At Once

Detecting customized moments and highlights from videos given natural
language (NL) user queries is an important but under-studied topic. One of the
challenges in pursuing this direction is the lack of annotated data. To address
this issue, we present the Query-based Video Highlights (QVHIGHLIGHTS) dataset.
It consists of over 10,000 YouTube videos, covering a wide range of topics,
from everyday activities and travel in lifestyle vlog videos to social and
political activities in news videos. Each video in the dataset is annotated
with: (1) a human-written free-form NL query, (2) relevant moments in the video
w.r.t. the query, and (3) five-point scale saliency scores for all
query-relevant clips. This comprehensive annotation enables us to develop and
evaluate systems that detect relevant moments as well as salient highlights for
diverse, flexible user queries. We also present a strong baseline for this
task, Moment-DETR, a transformer encoder-decoder model that views moment
retrieval as a direct set prediction problem, taking extracted video and query
representations as inputs and predicting moment coordinates and saliency scores
end-to-end. While our model does not utilize any human prior, we show that it
performs competitively when compared to well-engineered architectures. With
weakly supervised pretraining using ASR captions, MomentDETR substantially
outperforms previous methods. Lastly, we present several ablations and
visualizations of Moment-DETR. Data and code is publicly available at
this https URL

本篇研究提出了基于问题的视频亮点（QVHIGHLIGHTS）数据集，用于开发和评估系统以检测有关时刻以及突出亮点，同时使用 Moment-DETR，一个转换器编码器 - 解码器模型，将时刻检索视为直接预测问题，并且在利用 ASR 注释和进行弱监督预训练时表现出优异的性能。

QVHighlights: 通过自然语言查询识别视频中的时刻和亮点

QVHighlights: Detecting Moments and Highlights in Videos via Natural  Language Queries

In this work, we study how the finetuning stage in the pretrain-finetune
framework changes the behavior of a pretrained neural language generator. We
focus on the transformer encoder-decoder model for the open-domain dialogue
response generation task. Our major finding is that after standard finetuning,
the model forgets some of the important language generation skills acquired
during large-scale pretraining. We demonstrate the forgetting phenomenon
through a set of detailed behavior analysis from the perspectives of knowledge
transfer, context sensitivity, and function space projection. As a preliminary
attempt to alleviate the forgetting problem, we propose an intuitive finetuning
strategy named "mix-review". We find that mix-review effectively regularizes
the finetuning process, and the forgetting problem is alleviated to some
extent. Finally, we discuss interesting behavior of the resulting dialogue
model and its implications.

本研究探讨预训练 - 微调框架中的微调阶段如何改变预训练神经语言生成模型的行为，发现标准微调后模型遗忘了某些重要的语言生成技巧，提出了一种直观的微调策略 “混合回顾”，能够缓解模型遗忘现象，最终讨论了对话模型的有趣行为及其含义。