Short-term action anticipation (STA) in first-person videos is a challenging
task that involves understanding the next active object interactions and
predicting future actions. Existing action anticipation methods have primarily
focused on utilizing features extracted from video clips, but often overlooked
the importance of objects and their interactions. To this end, we propose a
novel approach that applies a guided attention mechanism between the objects,
and the spatiotemporal features extracted from video clips, enhancing the
motion and contextual information, and further decoding the object-centric and
motion-centric information to address the problem of STA in egocentric videos.
Our method, GANO (Guided Attention for Next active Objects) is a multi-modal,
end-to-end, single transformer-based network. The experimental results
performed on the largest egocentric dataset demonstrate that GANO outperforms
the existing state-of-the-art methods for the prediction of the next active
object label, its bounding box location, the corresponding future action, and
the time to contact the object. The ablation study shows the positive
contribution of the guided attention mechanism compared to other fusion
methods. Moreover, it is possible to improve the next active object location
and class label prediction results of GANO by just appending the learnable
object tokens with the region of interest embeddings.

该论文提出了一种新颖的方法，名为 GANO（Guided Attention for Next active Objects），该方法采用了在对象之间引导注意力机制和从视频剪辑中提取的时空特征，以增强运动和语境信息，并进一步解码面向对象和动态的信息，以解决视角视频中 STA 的问题。在最大的自我中心数据集上表现出比现有最先进方法更好的效果，可以预测下一个活动对象的标签，其边界框位置，相应的未来动作和接触对象的时间。

引导注意力增强基于对象的下一步自我中心动作预测

Enhancing Next Active Object-based Egocentric Action Anticipation with  Guided Attention

Current methods for video activity localisation over time assume implicitly
that activity temporal boundaries labelled for model training are determined
and precise. However, in unscripted natural videos, different activities mostly
transit smoothly, so that it is intrinsically ambiguous to determine in
labelling precisely when an activity starts and ends over time. Such
uncertainties in temporal labelling are currently ignored in model training,
resulting in learning mis-matched video-text correlation with poor
generalisation in test. In this work, we solve this problem by introducing
Elastic Moment Bounding (EMB) to accommodate flexible and adaptive activity
temporal boundaries towards modelling universally interpretable video-text
correlation with tolerance to underlying temporal uncertainties in pre-fixed
annotations. Specifically, we construct elastic boundaries adaptively by mining
and discovering frame-wise temporal endpoints that can maximise the alignment
between video segments and query sentences. To enable both more accurate
matching (segment content attention) and more robust localisation (segment
elastic boundaries), we optimise the selection of frame-wise endpoints subject
to segment-wise contents by a novel Guided Attention mechanism. Extensive
experiments on three video activity localisation benchmarks demonstrate
compellingly the EMB's advantages over existing methods without modelling
uncertainty.

本文提出 Elastic Moment Bounding (EMB) 和 guided attention 机制，解决了视频活动定位训练时时间标注的不确定性问题，提高了在自然视频中的准确性和鲁棒性。

带有时间边界不确定性的视频活动定位

Video Activity Localisation with Uncertainties in Temporal Boundary

The main goal of network pruning is imposing sparsity on the neural network
by increasing the number of parameters with zero value in order to reduce the
architecture size and the computational speedup. In most of the previous
research works, sparsity is imposed stochastically without considering any
prior knowledge of the weights distribution or other internal network
characteristics. Enforcing too much sparsity may induce accuracy drop due to
the fact that a lot of important elements might have been eliminated. In this
paper, we propose Guided Attention for Sparsity Learning (GASL) to achieve (1)
model compression by having less number of elements and speed-up; (2) prevent
the accuracy drop by supervising the sparsity operation via a guided attention
mechanism and (3) introduce a generic mechanism that can be adapted for any
type of architecture; Our work is aimed at providing a framework based on
interpretable attention mechanisms for imposing structured and non-structured
sparsity in deep neural networks. For Cifar-100 experiments, we achieved the
state-of-the-art sparsity level and 2.91x speedup with competitive accuracy
compared to the best method. For MNIST and LeNet architecture we also achieved
the highest sparsity and speedup level.

本研究提出了一种基于可解释的 attention mechanism 的网络修剪方法，能在保持准确率的前提下实现模型压缩和加速，可适用于各种结构的深度神经网络。实验证明，在 Cifar-100 数据集上取得了最高的稀疏度和加速比，与最佳方法相比准确率相当；在 MNIST 和 LeNet 架构上也取得了最高的稀疏度和加速比。

GASL: 深度神经网络稀疏学习的引导注意力

GASL: Guided Attention for Sparsity Learning in Deep Neural Networks

In this paper, we propose a novel end-to-end model, namely Single-Stage
Grounding network (SSG), to localize the referent given a referring expression
within an image. Different from previous multi-stage models which rely on
object proposals or detected regions, our proposed model aims to comprehend a
referring expression through one single stage without resorting to region
proposals as well as the subsequent region-wise feature extraction.
Specifically, a multimodal interactor is proposed to summarize the local region
features regarding the referring expression attentively. Subsequently, a
grounder is proposed to localize the referring expression within the given
image directly. For further improving the localization accuracy, a guided
attention mechanism is proposed to enforce the grounder to focus on the central
region of the referent. Moreover, by exploiting and predicting visual attribute
information, the grounder can further distinguish the referent objects within
an image and thereby improve the model performance. Experiments on RefCOCO,
RefCOCO+, and RefCOCOg datasets demonstrate that our proposed SSG without
relying on any region proposals can achieve comparable performance with other
advanced models. Furthermore, our SSG outperforms the previous models and
achieves the state-of-art performance on the ReferItGame dataset. More
importantly, our SSG is time efficient and can ground a referring expression in
a 416*416 image from the RefCOCO dataset in 25ms (40 referents per second) on
average with a Nvidia Tesla P40, accomplishing more than 9* speedups over the
existing multi-stage models.

本文提出了一种新的端到端模型 Single-Stage Grounding network（SSG），用于在图像内定位所指物体，通过多模态交互器和定位器来处理一个指代表达式，提出了引导注意机制和预测视觉属性信息来提高模型性能，并在 RefCOCO，RefCOCO + 和 RefCOCOg 数据集上进行了实验，结果表明设备效率高，能在很短时间内完成对所指物体的定位。