Video transformers have become the de facto standard for human action
recognition, yet their exclusive reliance on the RGB modality still limits
their adoption in certain domains. One such domain is Activities of Daily
Living (ADL), where RGB alone is not sufficient to distinguish between visually
similar actions, or actions observed from multiple viewpoints. To facilitate
the adoption of video transformers for ADL, we hypothesize that the
augmentation of RGB with human pose information, known for its sensitivity to
fine-grained motion and multiple viewpoints, is essential. Consequently, we
introduce the first Pose Induced Video Transformer: PI-ViT (or $\pi$-ViT), a
novel approach that augments the RGB representations learned by video
transformers with 2D and 3D pose information. The key elements of $\pi$-ViT are
two plug-in modules, 2D Skeleton Induction Module and 3D Skeleton Induction
Module, that are responsible for inducing 2D and 3D pose information into the
RGB representations. These modules operate by performing pose-aware auxiliary
tasks, a design choice that allows $\pi$-ViT to discard the modules during
inference. Notably, $\pi$-ViT achieves the state-of-the-art performance on
three prominent ADL datasets, encompassing both real-world and large-scale
RGB-D datasets, without requiring poses or additional computational overhead at
inference.

PI-ViT is a Pose Induced Video Transformer that augments RGB representations learned by video transformers with 2D and 3D pose information, achieving state-of-the-art performance for Activities of Daily Living (ADL) recognition on real-world and large-scale RGB-D datasets without additional computational overhead at inference.

仅加 $π$！用于理解日常生活活动的姿势引发视频 Transformer

Just Add $π$! Pose Induced Video Transformers for Understanding  Activities of Daily Living

Scene understanding is an active research area. Commercial depth sensors,
such as Kinect, have enabled the release of several RGB-D datasets over the
past few years which spawned novel methods in 3D scene understanding. More
recently with the launch of the LiDAR sensor in Apple's iPads and iPhones, high
quality RGB-D data is accessible to millions of people on a device they
commonly use. This opens a whole new era in scene understanding for the
Computer Vision community as well as app developers. The fundamental research
in scene understanding together with the advances in machine learning can now
impact people's everyday experiences. However, transforming these scene
understanding methods to real-world experiences requires additional innovation
and development. In this paper we introduce ARKitScenes. It is not only the
first RGB-D dataset that is captured with a now widely available depth sensor,
but to our best knowledge, it also is the largest indoor scene understanding
data released. In addition to the raw and processed data from the mobile
device, ARKitScenes includes high resolution depth maps captured using a
stationary laser scanner, as well as manually labeled 3D oriented bounding
boxes for a large taxonomy of furniture. We further analyze the usefulness of
the data for two downstream tasks: 3D object detection and color-guided depth
upsampling. We demonstrate that our dataset can help push the boundaries of
existing state-of-the-art methods and it introduces new challenges that better
represent real-world scenarios.

介绍了 ARKitScenes，这是第一个使用 RGB-D 数据集且具有广泛可用深度传感器的室内场景认知数据集，并分析了数据集的有用性，包括用于 3D 对象检测和颜色引导深度上采样。

ARKitScenes: 使用移动 RGB-D 数据的 3D 室内场景理解多样化真实世界数据集

ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene  Understanding Using Mobile RGB-D Data

Access to large, diverse RGB-D datasets is critical for training RGB-D scene
understanding algorithms. However, existing datasets still cover only a limited
number of views or a restricted scale of spaces. In this paper, we introduce
Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views
from 194,400 RGB-D images of 90 building-scale scenes. Annotations are provided
with surface reconstructions, camera poses, and 2D and 3D semantic
segmentations. The precise global alignment and comprehensive, diverse
panoramic set of views over entire buildings enable a variety of supervised and
self-supervised computer vision tasks, including keypoint matching, view
overlap prediction, normal prediction from color, semantic segmentation, and
region classification.

本文介绍 Matterport3D 数据集，包含 10,800 个全景视图，通过 RGB-D 图片提供了 90 个建筑尺度的场景的表面重建、相机姿态和 2D、3D 语义分割注释等，可用于多种计算机视觉任务，如关键点匹配、视图重叠预测、颜色预测等。