Combining different sensing modalities with multiple positions helps form a
unified perception and understanding of complex situations such as human
behavior. Hence, human activity recognition (HAR) benefits from combining
redundant and complementary information (Unimodal/Multimodal). Even so, it is
not an easy task. It requires a multidisciplinary approach, including expertise
in sensor technologies, signal processing, data fusion algorithms, and
domain-specific knowledge. This Ph.D. work employs sensing modalities such as
inertial, pressure (audio and atmospheric pressure), and textile capacitive
sensing for HAR. The scenarios explored are gesture and hand position tracking,
facial and head pattern recognition, and body posture and gesture recognition.
The selected wearable devices and sensing modalities are fully integrated with
machine learning-based algorithms, some of which are implemented in the
embedded device, on the edge, and tested in real-time.

综合不同的感知模态与多个位置有助于形成对人类行为等复杂情况的统一感知和理解，因此，人类活动识别受益于综合冗余和互补信息（单模态 / 多模态）。然而，这并不是一项容易的任务，它需要多学科的方法，包括传感器技术、信号处理、数据融合算法和领域特定知识。本博士研究使用惯性、压力（音频和大气压力）和纺织电容感应等感知模型进行人类活动识别。所探索的场景包括手势和手部位置跟踪、面部和头部模式识别以及身体姿势和手势识别。所选择的可穿戴设备和感知模型完全集成了基于机器学习的算法，其中一些实现在嵌入式设备上，并在实时环境中进行了测试。

可穿戴活动识别的单模态和多模态传感器融合

Unimodal and Multimodal Sensor Fusion for Wearable Activity Recognition

Leveraging sensing modalities across diverse spatial and temporal resolutions
can improve performance of robotic manipulation tasks. Multi-spatial resolution
sensing provides hierarchical information captured at different spatial scales
and enables both coarse and precise motions. Simultaneously multi-temporal
resolution sensing enables the agent to exhibit high reactivity and real-time
control. In this work, we propose a framework, MResT (Multi-Resolution
Transformer), for learning generalizable language-conditioned multi-task
policies that utilize sensing at different spatial and temporal resolutions
using networks of varying capacities to effectively perform real time control
of precise and reactive tasks. We leverage off-the-shelf pretrained
vision-language models to operate on low-frequency global features along with
small non-pretrained models to adapt to high frequency local feedback. Through
extensive experiments in 3 domains (coarse, precise and dynamic manipulation
tasks), we show that our approach significantly improves (2X on average) over
recent multi-task baselines. Further, our approach generalizes well to visual
and geometric variations in target objects and to varying interaction forces.

使用不同空间和时间分辨率的感知模式可以提高机器人操作任务的性能。本研究提出了一种名为 MResT（多分辨率变压器）的框架，利用具有不同容量的网络学习可推广的语言条件多任务策略，以有效地进行精确和反应迅速的实时控制。通过在 3 个领域（粗糙、精确和动态操作任务）进行大量实验，我们证明了我们的方法相对于最近的多任务基线显著改进（平均提升了 2 倍）。此外，我们的方法对目标物体的视觉和几何变化以及交互力的变化具有很好的普适性。

MResT：多分辨率感知实时控制的视觉语言模型

MResT: Multi-Resolution Sensing for Real-Time Control with  Vision-Language Models

Recent breakthroughs in deep learning and artificial intelligence
technologies have enabled numerous mobile applications. While traditional
computation paradigms rely on mobile sensing and cloud computing, deep learning
implemented on mobile devices provides several advantages. These advantages
include low communication bandwidth, small cloud computing resource cost, quick
response time, and improved data privacy. Research and development of deep
learning on mobile and embedded devices has recently attracted much attention.
This paper provides a timely review of this fast-paced field to give the
researcher, engineer, practitioner, and graduate student a quick grasp on the
recent advancements of deep learning on mobile devices. In this paper, we
discuss hardware architectures for mobile deep learning, including Field
Programmable Gate Arrays, Application Specific Integrated Circuit, and recent
mobile Graphic Processing Units. We present Size, Weight, Area and Power
considerations and their relation to algorithm optimizations, such as
quantization, pruning, compression, and approximations that simplify
computation while retaining performance accuracy. We cover existing systems and
give a state-of-the-industry review of TensorFlow, MXNet, Mobile AI Compute
Engine, and Paddle-mobile deep learning platform. We discuss resources for
mobile deep learning practitioners, including tools, libraries, models, and
performance benchmarks. We present applications of various mobile sensing
modalities to industries, ranging from robotics, healthcare and multi-media,
biometrics to autonomous drive and defense. We address the key deep learning
challenges to overcome, including low quality data, and small
training/adaptation data sets. In addition, the review provides numerous
citations and links to existing code bases implementing various technologies.

本文综述了移动设备上深度学习的研究进展，包括算法、硬件架构、感知模态等技术的最新发展，以及对应的工具、应用和挑战。