In this paper, we introduce a novel formulation for camera motion estimation
that integrates RGB-D images and inertial data through scene flow. Our goal is
to accurately estimate the camera motion in a rigid 3D environment, along with
the state of the inertial measurement unit (IMU). Our proposed method offers
the flexibility to operate as a multi-frame optimization or to marginalize
older data, thus effectively utilizing past measurements. To assess the
performance of our method, we conducted evaluations using both synthetic data
from the ICL-NUIM dataset and real data sequences from the OpenLORIS-Scene
dataset. Our results show that the fusion of these two sensors enhances the
accuracy of camera motion estimation when compared to using only visual data.

该研究介绍了一种新的相机运动估计方法，通过场景流集成 RGB-D 图像和惯性数据，旨在在刚性三维环境中准确估计相机运动和惯性测量单元（IMU）状态。通过对合成数据和真实数据进行评估，结果显示这两个传感器的融合相比仅使用视觉数据具有更高的相机运动估计精度。

基于 RGB-D 和惯性场景流的相机运动估计

Camera Motion Estimation from RGB-D-Inertial Scene Flow

A persistent trend in Deep Learning has been the applicability of machine
learning concepts to other areas than originally introduced for. As of today,
state-of-the-art activity recognition from wearable sensors relies on
classifiers being trained on fixed windows of data. Contrarily, video-based
Human Activity Recognition has followed a segment-based prediction approach,
localizing activity occurrences from start to end. This paper is the first to
systematically demonstrate the applicability of state-of-the-art TAL models for
wearable Human Activity Recongition (HAR) using raw inertial data as input. Our
results show that state-of-the-art TAL models are able to outperform popular
inertial models on 4 out of 6 wearable activity recognition benchmark datasets,
with improvements ranging as much as 25% in F1-score. Introducing the TAL
community's most popular metric to inertial-based HAR, namely mean Average
Precision, our analysis shows that TAL models are able to produce more coherent
segments along with an overall higher NULL-class accuracy across all datasets.
Being the first to provide such an analysis, the TAL community offers an
interesting new perspective to inertial-based HAR with yet to be explored
design choices and training concepts, which could be of significant value for
the inertial-based HAR community.

该论文首次系统地展示了最新的 TAL 模型在使用原始惯性数据作为输入进行可穿戴式人体活动识别（HAR）方面的适用性，并利用 4 个可穿戴活动识别基准数据集证明这些模型在 F1 分数上的提高幅度可达 25%。通过引入 TAL 社区的最流行度量标准 —— 均值平均精度，论文分析表明 TAL 模型能够产生更连贯的片段，并在所有数据集上具有更高的 NULL 类别准确率。作为首次提供这样的分析，TAL 社区为惯性式 HAR 提供了一个有待探索的设计选择和训练概念的有趣新视角，对惯性式 HAR 社区具有重要价值。

基于惯性的人体活动识别的时间动作定位

Temporal Action Localization for Inertial-based Human Activity  Recognition

Though research has shown the complementarity of camera- and inertial-based
data, datasets which offer both modalities remain scarce. In this paper we
introduce WEAR, a multimodal benchmark dataset for both vision- and
wearable-based Human Activity Recognition (HAR). The dataset comprises data
from 18 participants performing a total of 18 different workout activities with
untrimmed inertial (acceleration) and camera (egocentric video) data recorded
at 10 different outside locations. WEAR features a diverse set of activities
which are low in inter-class similarity and, unlike previous egocentric
datasets, not defined by human-object-interactions nor originate from
inherently distinct activity categories. Provided benchmark results reveal that
single-modality architectures have different strengths and weaknesses in their
prediction performance. Further, in light of the recent success of
transformer-based video action detection models, we demonstrate their
versatility by applying them in a plain fashion using vision, inertial and
combined (vision + inertial) features as input. Results show that vision
transformers are not only able to produce competitive results using only
inertial data, but also can function as an architecture to fuse both modalities
by means of simple concatenation, with the multimodal approach being able to
produce the highest average mAP, precision and close-to-best F1-scores. Up
until now, vision-based transformers have neither been explored in inertial nor
in multimodal human activity recognition, making our approach the first to do
so. The dataset and code to reproduce experiments is publicly available via:
mariusbock.github.io/wear

本文介绍了一个用于人类活动识别的多模态基准数据集 WEAR，其中包括了视觉和可穿戴设备的数据，有效地提高了各种算法的性能表现。此外，我们还展示了基于 Transformer 的视频动作检测模型在惯性和多模态人类活动识别方面的可行性，这是第一次使用此方法进行探索。