We present Compound Conditioned ControlNet, C3Net, a novel generative neural
architecture taking conditions from multiple modalities and synthesizing
multimodal contents simultaneously (e.g., image, text, audio). C3Net adapts the
ControlNet architecture to jointly train and make inferences on a
production-ready diffusion model and its trainable copies. Specifically, C3Net
first aligns the conditions from multi-modalities to the same semantic latent
space using modality-specific encoders based on contrastive training. Then, it
generates multimodal outputs based on the aligned latent space, whose semantic
information is combined using a ControlNet-like architecture called Control
C3-UNet. Correspondingly, with this system design, our model offers an improved
solution for joint-modality generation through learning and explaining
multimodal conditions instead of simply taking linear interpolations on the
latent space. Meanwhile, as we align conditions to a unified latent space,
C3Net only requires one trainable Control C3-UNet to work on multimodal
semantic information. Furthermore, our model employs unimodal pretraining on
the condition alignment stage, outperforming the non-pretrained alignment even
on relatively scarce training data and thus demonstrating high-quality compound
condition generation. We contribute the first high-quality tri-modal validation
set to validate quantitatively that C3Net outperforms or is on par with first
and contemporary state-of-the-art multimodal generation. Our codes and
tri-modal dataset will be released.

我们提出了 Compound Conditioned ControlNet（C3Net），这是一种新颖的生成神经架构，它从多种模态中获取条件并同时合成多模态内容（例如图像、文本、音频）。C3Net 将 ControlNet 架构适应到一个可生产的扩散模型及其可训练副本上。使用基于对比训练的模态特定编码器，C3Net 首先将多模态条件对齐到相同的语义潜空间，然后基于对齐的潜空间生成多模态输出，其语义信息使用称为 Control C3-UNet 的 ControlNet 类似架构进行组合。我们的模型通过学习和解释多模态条件而不仅仅在潜空间上进行线性插值，从而提供了一种改进的联合模态生成解决方案。此外，由于我们将条件对齐到统一的潜空间，C3Net 只需要一个可训练的 Control C3-UNet 来处理多模态语义信息。我们的模型在条件对齐阶段进行了单模态预训练，在相对稀缺的训练数据上优于未经预训练的对齐，从而展现了高质量的复合条件生成能力。我们提供了第一个高质量的三模态验证集，可定量验证 C3Net 在多模态生成方面优于或与最先进的方法相媲美。我们将发布我们的代码和三模态数据集。