This report presents ReLER submission to two tracks in the Ego4D Episodic
Memory Benchmark in CVPR 2023, including Natural Language Queries and Moment
Queries. This solution inherits from our proposed Action Sensitivity Learning
framework (ASL) to better capture discrepant information of frames. Further, we
incorporate a series of stronger video features and fusion strategies. Our
method achieves an average mAP of 29.34, ranking 1st in Moment Queries
Challenge, and garners 19.79 mean R1, ranking 2nd in Natural Language Queries
Challenge. Our code will be released.

本报告介绍了 ReLER 在 Ego4D 情节记忆基准测试中提交的两个轨道，包括自然语言查询和时刻查询。我们的解决方案继承了我们提出的行动敏感性学习框架（ASL），以更好地捕捉帧的差异信息。此外，我们整合了一系列更强大的视频特征和融合策略。我们的方法在时刻查询挑战赛中获得了 29.34 的平均 mAP，排名第 1，在自然语言查询挑战中获得了 19.79 的平均 R1，排名第 2。我们的代码将发布。

Ego4D 情景记忆挑战 2023 的行动敏感度学习

Action Sensitivity Learning for the Ego4D Episodic Memory Challenge 2023

Temporal action localization (TAL), which involves recognizing and locating
action instances, is a challenging task in video understanding. Most existing
approaches directly predict action classes and regress offsets to boundaries,
while overlooking the discrepant importance of each frame. In this paper, we
propose an Action Sensitivity Learning framework (ASL) to tackle this task,
which aims to assess the value of each frame and then leverage the generated
action sensitivity to recalibrate the training procedure. We first introduce a
lightweight Action Sensitivity Evaluator to learn the action sensitivity at the
class level and instance level, respectively. The outputs of the two branches
are combined to reweight the gradient of the two sub-tasks. Moreover, based on
the action sensitivity of each frame, we design an Action Sensitive Contrastive
Loss to enhance features, where the action-aware frames are sampled as positive
pairs to push away the action-irrelevant frames. The extensive studies on
various action localization benchmarks (i.e., MultiThumos, Charades,
Ego4D-Moment Queries v1.0, Epic-Kitchens 100, Thumos14 and ActivityNet1.3) show
that ASL surpasses the state-of-the-art in terms of average-mAP under multiple
types of scenarios, e.g., single-labeled, densely-labeled and egocentric.

本研究提出了一种 Action Sensitivity Learning 框架，旨在识别和定位动作示例，并通过轻量级的 Action Sensitivity Evaluator 和 Action Sensitive Contrastive Loss 实现对应的学习和训练。实验结果表明，该框架在多种场景下均优于现有技术，达到了最高的平均 mAP 值。