The main streams of human activity recognition (HAR) algorithms are developed
based on RGB cameras which are suffered from illumination, fast motion,
privacy-preserving, and large energy consumption. Meanwhile, the biologically
inspired event cameras attracted great interest due to their unique features,
such as high dynamic range, dense temporal but sparse spatial resolution, low
latency, low power, etc. As it is a newly arising sensor, even there is no
realistic large-scale dataset for HAR. Considering its great practical value,
in this paper, we propose a large-scale benchmark dataset to bridge this gap,
termed HARDVS, which contains 300 categories and more than 100K event
sequences. We evaluate and report the performance of multiple popular HAR
algorithms, which provide extensive baselines for future works to compare. More
importantly, we propose a novel spatial-temporal feature learning and fusion
framework, termed ESTF, for event stream based human activity recognition. It
first projects the event streams into spatial and temporal embeddings using
StemNet, then, encodes and fuses the dual-view representations using
Transformer networks. Finally, the dual features are concatenated and fed into
a classification head for activity prediction. Extensive experiments on
multiple datasets fully validated the effectiveness of our model. Both the
dataset and source code will be released on
https://github.com/Event-AHU/HARDVS.

本文提出了一个大规模的基准数据集 HARDVS 来填补基于事件相机的人类活动识别领域的数据空缺，同时还提出了一个名为 ESTF 的新的空间 - 时间特征学习和融合框架，结合了 STEMNET 和 Transformer 矩阵计算。结果表明，ESTF 能够有效地学习和推断人类活动。

使用动态视觉传感器重新审视人类活动识别

HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors

Many existing autonomous driving paradigms involve a multi-stage discrete
pipeline of tasks. To better predict the control signals and enhance user
safety, an end-to-end approach that benefits from joint spatial-temporal
feature learning is desirable. While there are some pioneering works on
LiDAR-based input or implicit design, in this paper we formulate the problem in
an interpretable vision-based setting. In particular, we propose a
spatial-temporal feature learning scheme towards a set of more representative
features for perception, prediction and planning tasks simultaneously, which is
called ST-P3. Specifically, an egocentric-aligned accumulation technique is
proposed to preserve geometry information in 3D space before the bird's eye
view transformation for perception; a dual pathway modeling is devised to take
past motion variations into account for future prediction; a temporal-based
refinement unit is introduced to compensate for recognizing vision-based
elements for planning. To the best of our knowledge, we are the first to
systematically investigate each part of an interpretable end-to-end
vision-based autonomous driving system. We benchmark our approach against
previous state-of-the-arts on both open-loop nuScenes dataset as well as
closed-loop CARLA simulation. The results show the effectiveness of our method.
Source code, model and protocol details are made publicly available at
this https URL

该研究旨在提高自主驾驶的性能和安全性，提出了一种解释性的基于视觉的空时特征学习方案，称为 ST-P3，通过端到端方式同时实现感知、预测和规划任务，并在开放循环 nuScenes 数据集和闭环 CARLA 仿真上验证了其优越性。

基于时空特征学习的端到端视觉自主驾驶

ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal  Feature Learning

Spatial-temporal feature learning is of vital importance for video emotion
recognition. Previous deep network structures often focused on macro-motion
which extends over long time scales, e.g., on the order of seconds. We believe
integrating structures capturing information about both micro- and macro-motion
will benefit emotion prediction, because human perceive both micro- and
macro-expressions. In this paper, we propose to combine micro- and macro-motion
features to improve video emotion recognition with a two-stream recurrent
network, named MIMAMO (Micro-Macro-Motion) Net. Specifically, smaller and
shorter micro-motions are analyzed by a two-stream network, while larger and
more sustained macro-motions can be well captured by a subsequent recurrent
network. Assigning specific interpretations to the roles of different parts of
the network enables us to make choice of parameters based on prior knowledge:
choices that turn out to be optimal. One of the important innovations in our
model is the use of interframe phase differences rather than optical flow as
input to the temporal stream. Compared with the optical flow, phase differences
require less computation and are more robust to illumination changes. Our
proposed network achieves state of the art performance on two video emotion
datasets, the OMG emotion dataset and the Aff-Wild dataset. The most
significant gains are for arousal prediction, for which motion information is
intuitively more informative. Source code is available at
this https URL

本文提出了一种称为 MIMAMO（Micro-Macro-Motion）网络的双流递归网络，将微动作和宏动作特征相结合，以改进视频情感识别，并在两个视频情感数据集上实现了最先进的性能。