By leveraging temporal dependency in video sequences, multi-frame human pose
estimation algorithms have demonstrated remarkable results in complicated
situations, such as occlusion, motion blur, and video defocus. These algorithms
are predominantly based on heatmaps, resulting in high computation and storage
requirements per frame, which limits their flexibility and real-time
application in video scenarios, particularly on edge devices. In this paper, we
develop an efficient and effective video-based human pose regression method,
which bypasses intermediate representations such as heatmaps and instead
directly maps the input to the output joint coordinates. Despite the inherent
spatial correlation among adjacent joints of the human pose, the temporal
trajectory of each individual joint exhibits relative independence. In light of
this, we propose a novel Decoupled Space-Time Aggregation network (DSTA) to
separately capture the spatial contexts between adjacent joints and the
temporal cues of each individual joint, thereby avoiding the conflation of
spatiotemporal dimensions. Concretely, DSTA learns a dedicated feature token
for each joint to facilitate the modeling of their spatiotemporal dependencies.
With the proposed joint-wise local-awareness attention mechanism, our method is
capable of efficiently and flexibly utilizing the spatial dependency of
adjacent joints and the temporal dependency of each joint itself. Extensive
experiments demonstrate the superiority of our method. Compared to previous
regression-based single-frame human pose estimation methods, DSTA significantly
enhances performance, achieving an 8.9 mAP improvement on PoseTrack2017.
Furthermore, our approach either surpasses or is on par with the
state-of-the-art heatmap-based multi-frame human pose estimation methods.
Project page: this https URL

通过综合利用视频序列的时间依赖性，我们开发了一种高效且有效的基于视频的人体姿态回归方法，绕过热图等中间表示，直接将输入映射到输出关节坐标。我们的方法利用了相邻关节的空间相关性和每个关节自身的时间相关性，通过提出的关节感知特征令牌机制来灵活地利用这些关联，从而显著提高了性能。

基于视频的人体姿态回归：通过解耦的时空聚合

Video-Based Human Pose Regression via Decoupled Space-Time Aggregation

Denoising diffusion probabilistic models that were initially proposed for
realistic image generation have recently shown success in various perception
tasks (e.g., object detection and image segmentation) and are increasingly
gaining attention in computer vision. However, extending such models to
multi-frame human pose estimation is non-trivial due to the presence of the
additional temporal dimension in videos. More importantly, learning
representations that focus on keypoint regions is crucial for accurate
localization of human joints. Nevertheless, the adaptation of the
diffusion-based methods remains unclear on how to achieve such objective. In
this paper, we present DiffPose, a novel diffusion architecture that formulates
video-based human pose estimation as a conditional heatmap generation problem.
First, to better leverage temporal information, we propose SpatioTemporal
Representation Learner which aggregates visual evidences across frames and uses
the resulting features in each denoising step as a condition. In addition, we
present a mechanism called Lookup-based MultiScale Feature Interaction that
determines the correlations between local joints and global contexts across
multiple scales. This mechanism generates delicate representations that focus
on keypoint regions. Altogether, by extending diffusion models, we show two
unique characteristics from DiffPose on pose estimation task: (i) the ability
to combine multiple sets of pose estimates to improve prediction accuracy,
particularly for challenging joints, and (ii) the ability to adjust the number
of iterative steps for feature refinement without retraining the model.
DiffPose sets new state-of-the-art results on three benchmarks: PoseTrack2017,
PoseTrack2018, and PoseTrack21.

DiffPose 是一种新颖的扩展扩散模型，将基于视频的人体姿势估计作为条件热图生成问题，并通过多组姿势估计的结合以及迭代步骤的调整来提高预测准确性，成功地在 PoseTrack2017、PoseTrack2018 和 PoseTrack21 三个基准测试上取得了新的最佳结果。

DiffPose：基于视频的人体姿势估计的时空扩散模型

DiffPose: SpatioTemporal Diffusion Model for Video-Based Human Pose  Estimation

Multi-frame human pose estimation has long been a compelling and fundamental
problem in computer vision. This task is challenging due to fast motion and
pose occlusion that frequently occur in videos. State-of-the-art methods strive
to incorporate additional visual evidences from neighboring frames (supporting
frames) to facilitate the pose estimation of the current frame (key frame). One
aspect that has been obviated so far, is the fact that current methods directly
aggregate unaligned contexts across frames. The spatial-misalignment between
pose features of the current frame and neighboring frames might lead to
unsatisfactory results. More importantly, existing approaches build upon the
straightforward pose estimation loss, which unfortunately cannot constrain the
network to fully leverage useful information from neighboring frames. To tackle
these problems, we present a novel hierarchical alignment framework, which
leverages coarse-to-fine deformations to progressively update a neighboring
frame to align with the current frame at the feature level. We further propose
to explicitly supervise the knowledge extraction from neighboring frames,
guaranteeing that useful complementary cues are extracted. To achieve this
goal, we theoretically analyzed the mutual information between the frames and
arrived at a loss that maximizes the task-relevant mutual information. These
allow us to rank No.1 in the Multi-frame Person Pose Estimation Challenge on
benchmark dataset PoseTrack2017, and obtain state-of-the-art performance on
benchmarks Sub-JHMDB and Pose-Track2018. Our code is released at
this https URL com/Pose-Group/FAMI-Pose, hoping that it will be useful to the
community.

本研究提出一种分层对齐框架，对多帧人体姿态估计问题进行了深入研究，利用视觉证据、知识提取和互信息等方法，在 PoseTrack2017、Sub-JHMDB 和 Pose-Track2018 等数据集上取得了最先进的性能表现。