Human Action Recognition (HAR) aims to understand human behavior and assign a
label to each action. It has a wide range of applications, and therefore has
been attracting increasing attention in the field of computer vision. Human
actions can be represented using various data modalities, such as RGB,
skeleton, depth, infrared, point cloud, event stream, audio, acceleration,
radar, and WiFi signal, which encode different sources of useful yet distinct
information and have various advantages depending on the application scenarios.
Consequently, lots of existing works have attempted to investigate different
types of approaches for HAR using various modalities. In this paper, we present
a comprehensive survey of recent progress in deep learning methods for HAR
based on the type of input data modality. Specifically, we review the current
mainstream deep learning methods for single data modalities and multiple data
modalities, including the fusion-based and the co-learning-based frameworks. We
also present comparative results on several benchmark datasets for HAR,
together with insightful observations and inspiring future research directions.

本文综述了基于不同数据形式的深度学习方法在人类动作识别方面的应用以及涉及 co-learning 和融合等技术，并对多个基础数据集进行了实验比较研究与探讨未来研究方向。

多模态数据中的人类动作识别：一篇综述

Human Action Recognition from Various Data Modalities: A Review

Current multi-object tracking and segmentation (MOTS) methods follow the
tracking-by-detection paradigm and adopt convolutions for feature extraction.
However, as affected by the inherent receptive field, convolution based feature
extraction inevitably mixes up the foreground features and the background
features, resulting in ambiguities in the subsequent instance association. In
this paper, we propose a highly effective method for learning instance
embeddings based on segments by converting the compact image representation to
un-ordered 2D point cloud representation. Our method generates a new
tracking-by-points paradigm where discriminative instance embeddings are
learned from randomly selected points rather than images. Furthermore, multiple
informative data modalities are converted into point-wise representations to
enrich point-wise features. The resulting online MOTS framework, named
PointTrack, surpasses all the state-of-the-art methods including 3D tracking
methods by large margins (5.4% higher MOTSA and 18 times faster over
MOTSFusion) with the near real-time speed (22 FPS). Evaluations across three
datasets demonstrate both the effectiveness and efficiency of our method.
Moreover, based on the observation that current MOTS datasets lack crowded
scenes, we build a more challenging MOTS dataset named APOLLO MOTS with higher
instance density. Both APOLLO MOTS and our codes are publicly available at
this https URL

本文提出了一种新的多对象跟踪方法，该方法利用无序的 2D 点云表示来生成实例嵌入，从随机选择的点中学习辨别实例嵌入，利用多种信息数据模态来丰富点特征，基于此建立了在线 MOTS 系统 PointTrack，在三个数据集上进行评估，并构建了一个更具挑战性的 MOTS 数据集 APOLLO MOTS。

基于点的分割，用于高效的在线多目标跟踪和分割

Segment as Points for Efficient Online Multi-Object Tracking and  Segmentation

The quest for algorithms that enable cognitive abilities is an important part
of machine learning. A common trait in many recently investigated
cognitive-like tasks is that they take into account different data modalities,
such as visual and textual input. In this paper we propose a novel and
generally applicable form of attention mechanism that learns high-order
correlations between various data modalities. We show that high-order
correlations effectively direct the appropriate attention to the relevant
elements in the different data modalities that are required to solve the joint
task. We demonstrate the effectiveness of our high-order attention mechanism on
the task of visual question answering (VQA), where we achieve state-of-the-art
performance on the standard VQA dataset.

本文提出了一种新颖且通用的注意力机制，可以学习不同数据模态之间的高阶相关性。 作者实验证明高阶相关性可以将适当的关注点引导到不同数据模态中的相关元素，来更好地解决联合任务，如视觉问答（VQA），在 VQA 标准数据集上实现了最先进的性能。