Existing vision-based action recognition is susceptible to occlusion and
appearance variations, while wearable sensors can alleviate these challenges by
capturing human motion with one-dimensional time-series signal. For the same
action, the knowledge learned from vision sensors and wearable sensors, may be
related and complementary. However, there exists significantly large modality
difference between action data captured by wearable-sensor and vision-sensor in
data dimension, data distribution and inherent information content. In this
paper, we propose a novel framework, named Semantics-aware Adaptive Knowledge
Distillation Networks (SAKDN), to enhance action recognition in vision-sensor
modality (videos) by adaptively transferring and distilling the knowledge from
multiple wearable sensors. The SAKDN uses multiple wearable-sensors as teacher
modalities and uses RGB videos as student modality. To preserve local temporal
relationship and facilitate employing visual deep learning model, we transform
one-dimensional time-series signals of wearable sensors to two-dimensional
images by designing a gramian angular field based virtual image generation
model. Then, we build a novel Similarity-Preserving Adaptive Multi-modal Fusion
Module to adaptively fuse intermediate representation knowledge from different
teacher networks. Finally, to fully exploit and transfer the knowledge of
multiple well-trained teacher networks to the student network, we propose a
novel Graph-guided Semantically Discriminative Mapping loss, which utilizes
graph-guided ablation analysis to produce a good visual explanation
highlighting the important regions across modalities and concurrently
preserving the interrelations of original data. Experimental results on
Berkeley-MHAD, UTD-MHAD and MMAct datasets well demonstrate the effectiveness
of our proposed SAKDN.

本文提出了一种名为 SAKDN 的新型框架，利用多个可穿戴传感器的知识在视觉传感器中增强动作识别，通过将可穿戴传感器的一维时间序列信号经过转换后进行多模态知识蒸馏和自适应融合，利用图形引导抑制映射损失，提高动作识别的效果。实验结果证实了本文 SAKDN 框架的有效性。

面向传感器 - 视觉动作识别的语义感知自适应知识蒸馏

Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision  Action Recognition

The success of deep learning in vision can be attributed to: (a) models with
high capacity; (b) increased computational power; and (c) availability of
large-scale labeled data. Since 2012, there have been significant advances in
representation capabilities of the models and computational capabilities of
GPUs. But the size of the biggest dataset has surprisingly remained constant.
What will happen if we increase the dataset size by 10x or 100x? This paper
takes a step towards clearing the clouds of mystery surrounding the
relationship between `enormous data' and visual deep learning. By exploiting
the JFT-300M dataset which has more than 375M noisy labels for 300M images, we
investigate how the performance of current vision tasks would change if this
data was used for representation learning. Our paper delivers some surprising
(and some expected) findings. First, we find that the performance on vision
tasks increases logarithmically based on volume of training data size. Second,
we show that representation learning (or pre-training) still holds a lot of
promise. One can improve performance on many vision tasks by just training a
better base model. Finally, as expected, we present new state-of-the-art
results for different vision tasks including image classification, object
detection, semantic segmentation and human pose estimation. Our sincere hope is
that this inspires vision community to not undervalue the data and develop
collective efforts in building larger datasets.

通过 JFT-300M 数据集进行实验研究，发现模型的训练数据量与视觉任务的表现成对数关系，预训练模型仍具有很大的潜力，同时通过该数据集在不同视觉任务上得到了新的最优表现。希望研究者重视数据集构建和扩大规模的重要性。