Large Multimodal Models (e.g., GPT-4, Gemini, Chameleon) have evolved into powerful tools with millions of users. However, they remain generic models and lack personalized knowledge of specific user concepts. Previous work has explored personalization for text generation, yet it remains unclear how these methods can be adapted to new modalities, such as image generation. In this paper, we introduce Yo'Chameleon, the first attempt to study personalization for large multimodal models. Given 3-5 images of a particular concept, Yo'Chameleon leverages soft-prompt tuning to embed subject-specific information to (i) answer questions about the subject and (ii) recreate pixel-level details to produce images of the subject in new contexts. Yo'Chameleon is trained with (i) a self-prompting optimization mechanism to balance performance across multiple modalities, and (ii) a ``soft-positive" image generation approach to enhance image quality in a few-shot setting.

本研究解决了大型多模态模型在个性化知识上的不足，特别是在图像生成方面的应用。提出的Yo'Chameleon利用软提示调整技术，能够针对特定概念结合用户信息进行多模态回答和图像生成。研究表明，该方法能够在少量样本条件下提高图像质量，并在多个模态中保持良好的性能。

YoChameleon：个性化视觉和语言生成