We introduce JointNet, a novel neural network architecture for modeling the joint distribution of images and an additional dense modality (e.g., depth maps). JointNet is extended from a pre-trained text-to-image diffusion model, where a copy of the original network is created for the new dense modality branch and is densely connected with the RGB branch. The RGB branch is locked during network fine-tuning, which enables efficient learning of the new modality distribution while maintaining the strong generalization ability of the large-scale pre-trained diffusion model. We demonstrate the effectiveness of JointNet by using RGBD diffusion as an example and through extensive experiments, showcasing its applicability in a variety of applications, including joint RGBD generation, dense depth prediction, depth-conditioned image generation, and coherent tile-based 3D panorama generation.

介绍了JointNet，一种用于建模图像和其他密集模态（例如深度图）联合分布的新型神经网络架构，并通过RGBD扩散作为示例以及广泛的实验证明了其在众多应用中的适用性，包括联合RGBD生成、密集深度预测、深度条件图像生成和一致的基于瓷砖的3D全景生成。

JointNet：基于文本图像扩散的密集分布建模