Recent text-to-image customization works have been proven successful in
generating images of given concepts by fine-tuning the diffusion models on a
few examples. However, these methods tend to overfit the concepts, resulting in
failure to create the concept under multiple conditions (e.g. headphone is
missing when generating a <sks> dog wearing a headphone'). Interestingly, we
notice that the base model before fine-tuning exhibits the capability to
compose the base concept with other elements (e.g. a dog wearing a headphone)
implying that the compositional ability only disappears after personalization
tuning. Inspired by this observation, we present ClassDiffusion, a simple
technique that leverages a semantic preservation loss to explicitly regulate
the concept space when learning the new concept. Despite its simplicity, this
helps avoid semantic drift when fine-tuning on the target concepts. Extensive
qualitative and quantitative experiments demonstrate that the use of semantic
preservation loss effectively improves the compositional abilities of the
fine-tune models. In response to the ineffective evaluation of CLIP-T metrics,
we introduce BLIP2-T metric, a more equitable and effective evaluation metric
for this particular domain. We also provide in-depth empirical study and
theoretical analysis to better understand the role of the proposed loss.
Lastly, we also extend our ClassDiffusion to personalized video generation,
demonstrating its flexibility.

最近的文本到图像定制方法在少量示例的基础上通过调整扩散模型已被证明成功生成给定概念的图像。然而，这些方法往往对概念过拟合，在多种条件下（例如，“在生成戴耳机的狗时耳机丢失”）无法创建概念。我们注意到，在微调之前的基准模型表现出与其他元素（例如，“戴耳机的狗”）组合的能力，暗示了这种组合能力仅在个性化调整后消失。受此观察的启发，我们提出了 ClassDiffusion，这是一种利用语义保持损失在学习新概念时明确调控概念空间的简单技术。尽管其简单性，这有助于在目标概念上进行微调时避免语义漂移。广泛的定性和定量实验证明了使用语义保持损失有效地改善了微调模型的组合能力。针对 CLIP-T 指标的无效评估，我们提出了 BLIP2-T 指标，这是一种针对这个特定领域更公正和有效的评估指标。我们还对所提出的损失的角色进行了深入实证研究和理论分析。最后，我们还将 ClassDiffusion 扩展到个性化视频生成，展示了其灵活性。

ClassDiffusion：使用明确的类别引导实现更加一致的个性化调优

ClassDiffusion: More Aligned Personalization Tuning with Explicit Class  Guidance

Data collection has become an increasingly important problem in robotic
manipulation, yet there still lacks much understanding of how to effectively
collect data to facilitate broad generalization. Recent works on large-scale
robotic data collection typically vary a wide range of environmental factors
during data collection, such as object types and table textures. While these
works attempt to cover a diverse variety of scenarios, they do not explicitly
account for the possible compositional abilities of policies trained on the
data. If robot policies are able to compose different environmental factors of
variation (e.g., object types, table heights) from their training data to
succeed when encountering unseen factor combinations, then we can exploit this
to avoid collecting data for situations that composition would address. To
investigate this possibility, we conduct thorough empirical studies both in
simulation and on a real robot that compare data collection strategies and
assess whether visual imitation learning policies can compose environmental
factors. We find that policies do exhibit composition, although leveraging
prior robotic datasets is critical for this on a real robot. We use these
insights to provide better practices for in-domain data collection by proposing
data collection strategies that exploit composition, which can induce better
generalization than naive approaches for the same amount of effort during data
collection. We further demonstrate that a real robot policy trained on data
from such a strategy achieves a success rate of 77.5% when transferred to
entirely new environments that encompass unseen combinations of environmental
factors, whereas policies trained using data collected without accounting for
environmental variation fail to transfer effectively, with a success rate of
only 2.5%. We provide videos at this http URL

通过大规模机器人数据收集，研究表明，在考虑环境因素的情况下利用组合性训练数据，可以更好地提高机器人政策的泛化能力，从而避免收集针对特定情况的数据。