With the ability to generate high-quality images, text-to-image (T2I) models
can be exploited for creating inappropriate content. To prevent misuse,
existing safety measures are either based on text blacklists, which can be
easily circumvented, or harmful content classification, requiring large
datasets for training and offering low flexibility. Hence, we propose Latent
Guard, a framework designed to improve safety measures in text-to-image
generation. Inspired by blacklist-based approaches, Latent Guard learns a
latent space on top of the T2I model's text encoder, where it is possible to
check the presence of harmful concepts in the input text embeddings. Our
proposed framework is composed of a data generation pipeline specific to the
task using large language models, ad-hoc architectural components, and a
contrastive learning strategy to benefit from the generated data. The
effectiveness of our method is verified on three datasets and against four
baselines. Code and data will be shared at
this https URL

通过在文本与图像生成模型的文本编码器上学习潜空间，Latent Guard 提出了一种用于改进文本与图像生成中安全性措施的框架，能够检测输入文本嵌入中存在的有害概念。