In this work, we aim to improve the 3D reasoning ability of Transformers in
multi-view 3D human pose estimation. Recent works have focused on end-to-end
learning-based transformer designs, which struggle to resolve geometric
information accurately, particularly during occlusion. Instead, we propose a
novel hybrid model, MVGFormer, which has a series of geometric and appearance
modules organized in an iterative manner. The geometry modules are
learning-free and handle all viewpoint-dependent 3D tasks geometrically which
notably improves the model's generalization ability. The appearance modules are
learnable and are dedicated to estimating 2D poses from image signals
end-to-end which enables them to achieve accurate estimates even when occlusion
occurs, leading to a model that is both accurate and generalizable to new
cameras and geometries. We evaluate our approach for both in-domain and
out-of-domain settings, where our model consistently outperforms
state-of-the-art methods, and especially does so by a significant margin in the
out-of-domain setting. We will release the code and models:
this https URL

我们的研究旨在提高 Transformer 的三维推理能力，在多视图三维人体姿势估计中。通过引入一种新的混合模型，MVGFormer，该模型采用一系列的几何和外观模块，以迭代方式组织。其中，几何模块通过几何方式处理依赖于视角的三维任务，显著改善了模型的泛化能力。外观模块是可学习的，并且专门用于从图像信号中端到端地估计二维姿势，即使在出现遮挡时也能获得准确的估计结果，从而实现了准确性和适用于新的摄像机和几何形状的模型。经过领域内外的评估，我们的模型始终优于最先进的方法，特别是在领域外的情况下有明显的优势。我们将发布代码和模型：this URL

多视角几何变换器用于 3D 人体姿态估计

Multiple View Geometry Transformers for 3D Human Pose Estimation

Recent advancements in differentiable rendering and 3D reasoning have driven
exciting results in novel view synthesis from a single image. Despite realistic
results, methods are limited to relatively small view change. In order to
synthesize immersive scenes, models must also be able to extrapolate. We
present an approach that fuses 3D reasoning with autoregressive modeling to
outpaint large view changes in a 3D-consistent manner, enabling scene
synthesis. We demonstrate considerable improvement in single image large-angle
view synthesis results compared to a variety of methods and possible variants
across simulated and real datasets. In addition, we show increased 3D
consistency compared to alternative accumulation methods. Project website:
this https URL

使用可微分渲染和 3D 推理相结合的自回归建模方法，实现了单幅图像的大视角高度一致性生成，比其他方法在视图生成和 3D 一致性方面具有显著的改进。

PixelSynth: 从单张图像生成 3D 一致的体验

PixelSynth: Generating a 3D-Consistent Experience from a Single Image

While recent generative models for 2D images achieve impressive visual
results, they clearly lack the ability to perform 3D reasoning. This heavily
restricts the degree of control over generated objects as well as the possible
applications of such models. In this work, we bridge this gap by leveraging
recent advances in differentiable rendering. We design a framework that can
generate triangle meshes and associated high-resolution texture maps, using
only 2D supervision from single-view natural images. A key contribution of our
work is the encoding of the mesh and texture as 2D representations, which are
semantically aligned and can be easily modeled by a 2D convolutional GAN. We
demonstrate the efficacy of our method on Pascal3D+ Cars and CUB, both in an
unconditional setting and in settings where the model is conditioned on class
labels, attributes, and text. Finally, we propose an evaluation methodology
that assesses the mesh and texture quality separately.

本文提出使用不同 iable rendering 技术生成三角网格和相关高分辨率纹理贴图的框架，以单视角自然图像为 2D 监督，并将网格和纹理编码为 2D 表示，以便使用 2D 卷积 GAN 进行建模，并在 Pascal3D + Cars 和 CUB 上展示了方法的有效性及其评估方法。

纹理三维网格卷积生成

Convolutional Generation of Textured 3D Meshes

We propose a CNN-based approach for multi-camera markerless motion capture of
the human body. Unlike existing methods that first perform pose estimation on
individual cameras and generate 3D models as post-processing, our approach
makes use of 3D reasoning throughout a multi-stage approach. This novelty
allows us to use provisional 3D models of human pose to rethink where the
joints should be located in the image and to recover from past mistakes. Our
principled refinement of 3D human poses lets us make use of image cues, even
from images where we previously misdetected joints, to refine our estimates as
part of an end-to-end approach. Finally, we demonstrate how the high-quality
output of our multi-camera setup can be used as an additional training source
to improve the accuracy of existing single camera models.

通过多相机无标记运动捕捉，使用基于 CNN 的方法进行人体运动捕捉，利用三维推理提高关节点定位准确性，并利用高质量的多相机训练数据来提高现有单相机模型的准确性。