Natural language serves as a common and straightforward control signal for
humans to interact seamlessly with machines. Recognizing the importance of this
interface, the machine learning community is investing considerable effort in
generating data that is semantically coherent with textual instructions. While
strides have been made in text-to-data generation spanning image editing, audio
synthesis, video creation, and beyond, low-resource areas characterized by
expensive annotations or complex data structures, such as molecules, motion
dynamics, and time series, often lack textual labels. This deficiency impedes
supervised learning, thereby constraining the application of advanced
generative models for text-to-data tasks. In response to these challenges in
the low-resource scenario, we propose Text2Data, a novel approach that utilizes
unlabeled data to understand the underlying data distribution through an
unsupervised diffusion model. Subsequently, it undergoes controllable
finetuning via a novel constraint optimization-based learning objective that
ensures controllability and effectively counteracts catastrophic forgetting.
Comprehensive experiments demonstrate that Text2Data is able to achieve
enhanced performance regarding controllability across various modalities,
including molecules, motions and time series, when compared to existing
baselines.

通过无标注数据进行无监督扩散建模的 Text2Data 方法在低资源环境中能够实现跨多种模态的可控性，并具备比现有基线模型更好的性能。