Text-to-image diffusion models understand spatial relationship between
objects, but do they represent the true 3D structure of the world from only 2D
supervision? We demonstrate that yes, 3D knowledge is encoded in 2D image
diffusion models like Stable Diffusion, and we show that this structure can be
exploited for 3D vision tasks. Our method, Viewpoint Neural Textual Inversion
(ViewNeTI), controls the 3D viewpoint of objects in generated images from
frozen diffusion models. We train a small neural mapper to take camera
viewpoint parameters and predict text encoder latents; the latents then
condition the diffusion generation process to produce images with the desired
camera viewpoint.
ViewNeTI naturally addresses Novel View Synthesis (NVS). By leveraging the
frozen diffusion model as a prior, we can solve NVS with very few input views;
we can even do single-view novel view synthesis. Our single-view NVS
predictions have good semantic details and photorealism compared to prior
methods. Our approach is well suited for modeling the uncertainty inherent in
sparse 3D vision problems because it can efficiently generate diverse samples.
Our view-control mechanism is general, and can even change the camera view in
images generated by user-defined prompts.

通过冻结的扩散模型，我们的方法 ViewNeTI 可以控制生成图像中物体的三维观点，以解决新颖视角合成的问题，并且拥有良好的语义细节和照片般逼真的单视图新颖视角合成预测。

视角文本倒置：通过预训练 2D 扩散模型释放新颖视图综合

Viewpoint Textual Inversion: Unleashing Novel View Synthesis with  Pretrained 2D Diffusion Models

The deficiency of 3D segmentation labels is one of the main obstacles to
effective point cloud segmentation, especially for scenes in the wild with
varieties of different objects. To alleviate this issue, we propose a novel
deep graph convolutional network-based framework for large-scale semantic scene
segmentation in point clouds with sole 2D supervision. Different with numerous
preceding multi-view supervised approaches focusing on single object point
clouds, we argue that 2D supervision is capable of providing sufficient
guidance information for training 3D semantic segmentation models of natural
scene point clouds while not explicitly capturing their inherent structures,
even with only single view per training sample. Specifically, a Graph-based
Pyramid Feature Network (GPFN) is designed to implicitly infer both global and
local features of point sets and an Observability Network (OBSNet) is
introduced to further solve object occlusion problem caused by complicated
spatial relations of objects in 3D scenes. During the projection process,
perspective rendering and semantic fusion modules are proposed to provide
refined 2D supervision signals for training along with a 2D-3D joint
optimization strategy. Extensive experimental results demonstrate the
effectiveness of our 2D supervised framework, which achieves comparable results
with the state-of-the-art approaches trained with full 3D labels, for semantic
point cloud segmentation on the popular SUNCG synthetic dataset and S3DIS
real-world dataset.

提出了一种基于 2D 监督的新型深度图卷积网络框架，用于大规模语义场景点云的分割，通过 Graph-based Pyramid Feature Network 和 Observability Network 可以隐式推断点集的全局和局部特征，并提出了一个 2D-3D 联合优化策略用于优化投影过程中的语义信息的加入， 即使只有单个训练样本的情况下，也能够进行自然场景下的 3D 分割训练，扩展实验在 SUNCG 和 S3DIS 数据集上证明了这种 2D 监督框架在语义点云分割方面的有效性。

野外场景三维图结构点云的弱监督语义分割

Weakly Supervised Semantic Segmentation in 3D Graph-Structured Point  Clouds of Wild Scenes

Estimating 3D hand pose from 2D images is a difficult, inverse problem due to
the inherent scale and depth ambiguities. Current state-of-the-art methods
train fully supervised deep neural networks with 3D ground-truth data. However,
acquiring 3D annotations is expensive, typically requiring calibrated
multi-view setups or labor intensive manual annotations. While annotations of
2D keypoints are much easier to obtain, how to efficiently leverage such
weakly-supervised data to improve the task of 3D hand pose prediction remains
an important open question. The key difficulty stems from the fact that direct
application of additional 2D supervision mostly benefits the 2D proxy objective
but does little to alleviate the depth and scale ambiguities. Embracing this
challenge we propose a set of novel losses. We show by extensive experiments
that our proposed constraints significantly reduce the depth ambiguity and
allow the network to more effectively leverage additional 2D annotated images.
For example, on the challenging freiHAND dataset using additional 2D annotation
without our proposed biomechanical constraints reduces the depth error by only
$15\%$, whereas the error is reduced significantly by $50\%$ when the proposed
biomechanical constraints are used.

本研究提出了一组创新的损失函数用于从 2D 图像中估计 3D 手势姿势，相较于仅仅使用 2D 标注数据，使用提出的生物力学约束显著减少了深度误差，提高了神经网络的准确性。