We present a novel method for recovering the absolute pose and shape of a
human in a pre-scanned scene given a single image. Unlike previous methods that
perform sceneaware mesh optimization, we propose to first estimate absolute
position and dense scene contacts with a sparse 3D CNN, and later enhance a
pretrained human mesh recovery network by cross-attention with the derived 3D
scene cues. Joint learning on images and scene geometry enables our method to
reduce the ambiguity caused by depth and occlusion, resulting in more
reasonable global postures and contacts. Encoding scene-aware cues in the
network also allows the proposed method to be optimization-free, and opens up
the opportunity for real-time applications. The experiments show that the
proposed network is capable of recovering accurate and physically-plausible
meshes by a single forward pass and outperforms state-of-the-art methods in
terms of both accuracy and speed.

提出了一种新方法，通过使用稀疏的 3D CNN 估计场景的绝对位置和密集的场景接触，并通过与其相关的 3D 场景线索的交互关注来加强预训练的人类 mesh 回收网络的关节学习，进而提高每个正向遍历的准确性和速度。

学习三维场景中的人体网格恢复

Learning Human Mesh Recovery in 3D Scenes

We present XFormer, a novel human mesh and motion capture method that
achieves real-time performance on consumer CPUs given only monocular images as
input. The proposed network architecture contains two branches: a keypoint
branch that estimates 3D human mesh vertices given 2D keypoints, and an image
branch that makes predictions directly from the RGB image features. At the core
of our method is a cross-modal transformer block that allows information to
flow across these two branches by modeling the attention between 2D keypoint
coordinates and image spatial features. Our architecture is smartly designed,
which enables us to train on various types of datasets including images with
2D/3D annotations, images with 3D pseudo labels, and motion capture datasets
that do not have associated images. This effectively improves the accuracy and
generalization ability of our system. Built on a lightweight backbone
(MobileNetV3), our method runs blazing fast (over 30fps on a single CPU core)
and still yields competitive accuracy. Furthermore, with an HRNet backbone,
XFormer delivers state-of-the-art performance on Huamn3.6 and 3DPW datasets.

XFormer 是一种新型的人体网格和动作捕捉方法，能够仅用单目图像即可实现在普通 CPU 上的实时性能，并能够高效地训练于多种数据集，其关键在于跨模态的 transformer block。

XFormer: 快速准确的单目三维人体捕捉

XFormer: Fast and Accurate Monocular 3D Body Capture

Inspired by the success of volumetric 3D pose estimation, some recent human
mesh estimators propose to estimate 3D skeletons as intermediate
representations, from which, the dense 3D meshes are regressed by exploiting
the mesh topology. However, body shape information is lost in extracting
skeletons, leading to mediocre performance. The advanced motion capture systems
solve the problem by placing dense physical markers on the body surface, which
allows to extract realistic meshes from their non-rigid motions. However, they
cannot be applied to wild images without markers. In this work, we present an
intermediate representation, named virtual markers, which learns 64 landmark
keypoints on the body surface based on the large-scale mocap data in a
generative style, mimicking the effects of physical markers. The virtual
markers can be accurately detected from wild images and can reconstruct the
intact meshes with realistic shapes by simple interpolation. Our approach
outperforms the state-of-the-art methods on three datasets. In particular, it
surpasses the existing methods by a notable margin on the SURREAL dataset,
which has diverse body shapes. Code is available at
this https URL

本文提出了基于虚拟标记的中间表示方法，通过学习 64 个身体表面关键点来重构逼真的 3D 网格，精度高于现有方法。

从虚拟标记估计 3D 人体网格

3D Human Mesh Estimation from Virtual Markers

Pose transfer has been studied for decades, in which the pose of a source
mesh is applied to a target mesh. Particularly in this paper, we are interested
in transferring the pose of source human mesh to deform the target human mesh,
while the source and target meshes may have different identity information.
Traditional studies assume that the paired source and target meshes are existed
with the point-wise correspondences of user annotated landmarks/mesh points,
which requires heavy labelling efforts. On the other hand, the generalization
ability of deep models is limited, when the source and target meshes have
different identities. To break this limitation, we proposes the first neural
pose transfer model that solves the pose transfer via the latest technique for
image style transfer, leveraging the newly proposed component -- spatially
adaptive instance normalization. Our model does not require any correspondences
between the source and target meshes. Extensive experiments show that the
proposed model can effectively transfer deformation from source to target
meshes, and has good generalization ability to deal with unseen identities or
poses of meshes. Code is available at
this https URL .

本文提出了一种基于深度学习的神经模型，应用最新的空间自适应实例规范化技术来解决源网格姿势传递到目标网格上。通过实验证明，所提出的模型可以有效地传递源到目标网格的变形，并具有很好的推广能力来处理未见过的网格姿势。