3D human generation is increasingly significant in various applications.
However, the direct use of 2D generative methods in 3D generation often results
in significant loss of local details, while methods that reconstruct geometry
from generated images struggle with global view consistency. In this work, we
introduce Joint2Human, a novel method that leverages 2D diffusion models to
generate detailed 3D human geometry directly, ensuring both global structure
and local details. To achieve this, we employ the Fourier occupancy field (FOF)
representation, enabling the direct production of 3D shapes as preliminary
results using 2D generative models. With the proposed high-frequency enhancer
and the multi-view recarving strategy, our method can seamlessly integrate the
details from different views into a uniform global shape.To better utilize the
3D human prior and enhance control over the generated geometry, we introduce a
compact spherical embedding of 3D joints. This allows for effective application
of pose guidance during the generation process. Additionally, our method is
capable of generating 3D humans guided by textual inputs. Our experimental
results demonstrate the capability of our method to ensure global structure,
local details, high resolution, and low computational cost, simultaneously.
More results and code can be found on our project page at
this http URL

通过 2D 扩散模型直接生成详细的 3D 人体几何结构，利用 Fourier 占据场表示法产生 3D 形状，并使用紧凑的球面嵌入 3D 关节，实现更好的控制和全局结构的生成，同时保持局部细节、高分辨率和低计算成本。

通过紧凑球面嵌入三维关节实现高质量三维人体生成

Joint2Human: High-quality 3D Human Generation via Compact Spherical  Embedding of 3D Joints

360$^\circ$ videos convey holistic views for the surroundings of a scene. It
provides audio-visual cues beyond pre-determined normal field of views and
displays distinctive spatial relations on a sphere. However, previous benchmark
tasks for panoramic videos are still limited to evaluate the semantic
understanding of audio-visual relationships or spherical spatial property in
surroundings. We propose a novel benchmark named Pano-AVQA as a large-scale
grounded audio-visual question answering dataset on panoramic videos. Using
5.4K 360$^\circ$ video clips harvested online, we collect two types of novel
question-answer pairs with bounding-box grounding: spherical spatial relation
QAs and audio-visual relation QAs. We train several transformer-based models
from Pano-AVQA, where the results suggest that our proposed spherical spatial
embeddings and multimodal training objectives fairly contribute to a better
semantic understanding of the panoramic surroundings on the dataset.

本文提出了 Pano-AVQA 基准测试用于评估全景视频中音频 - 视觉关系和球形空间关系的语义理解。使用在线获取的 5.4K 个视频剪辑，收集了两种类型的新型问题 - 答案对。通过球形空间嵌入和多模态训练目标，使用多个基于 Transformer 的模型从 Pano-AVQA 中进行训练，结果表明我们的提出的球形空间嵌入和多模态训练目标对数据集上全景环境的语义理解有很好的帮助。