Personalized text-to-image generation has attracted unprecedented attention
in the recent few years due to its unique capability of generating
highly-personalized images via using the input concept dataset and novel
textual prompt. However, previous methods solely focus on the performance of
the reconstruction task, degrading its ability to combine with different
textual prompt. Besides, optimizing in the high-dimensional embedding space
usually leads to unnecessary time-consuming training process and slow
convergence. To address these issues, we propose an efficient method to explore
the target embedding in a textual subspace, drawing inspiration from the
self-expressiveness property. Additionally, we propose an efficient selection
strategy for determining the basis vectors of the textual subspace. The
experimental evaluations demonstrate that the learned embedding can not only
faithfully reconstruct input image, but also significantly improves its
alignment with novel input textual prompt. Furthermore, we observe that
optimizing in the textual subspace leads to an significant improvement of the
robustness to the initial word, relaxing the constraint that requires users to
input the most relevant initial word. Our method opens the door to more
efficient representation learning for personalized text-to-image generation.

我们提出了一种有效的方法来在文本子空间中探索目标嵌入，借鉴了自我表现性特性，并提出了一种有效的选择策略来确定文本子空间的基向量。实验评估表明，所学习的嵌入不仅能够忠实地重构输入图像，还能显著改善其与新的输入文本提示的对齐性。此外，我们观察到在文本子空间中优化能够显著提高对初始词的鲁棒性，从而放宽了用户需要输入最相关初始词的约束。我们的方法为个性化文本到图像生成的更高效的表示学习打开了大门。

通过利用文本子空间提高高效个性化文本到图像生成

Efficient Personalized Text-to-image Generation by Leveraging Textual  Subspace

Masked visual modeling has attracted much attention due to its promising
potential in learning generalizable representations. Typical approaches urge
models to predict specific contents of masked tokens, which can be intuitively
considered as teaching a student (the model) to solve given problems
(predicting masked contents). Under such settings, the performance is highly
correlated with mask strategies (the difficulty of provided problems). We argue
that it is equally important for the model to stand in the shoes of a teacher
to produce challenging problems by itself. Intuitively, patches with high
values of reconstruction loss can be regarded as hard samples, and masking
those hard patches naturally becomes a demanding reconstruction task. To
empower the model as a teacher, we propose Hard Patches Mining (HPM),
predicting patch-wise losses and subsequently determining where to mask.
Technically, we introduce an auxiliary loss predictor, which is trained with a
relative objective to prevent overfitting to exact loss values. Also, to
gradually guide the training procedure, we propose an easy-to-hard mask
strategy. Empirically, HPM brings significant improvements under both image and
video benchmarks. Interestingly, solely incorporating the extra loss prediction
objective leads to better representations, verifying the efficacy of
determining where is hard to reconstruct. The code is available at
this https URL

通过引入一个辅助损失预测器和一个逐渐引导训练过程的易难遮罩策略，我们提出了 Hard Patches Mining (HPM) 方法来增强模型作为教师的能力，通过预测基于补丁的损失并确定遮罩的位置，从而显著改进了图像和视频的基准性能，并验证了确定难以重建的位置对于更好的表示的有效性。

通过硬补丁挖掘引导掩蔽视觉建模

Bootstrap Masked Visual Modeling via Hard Patches Mining

Prior works have demonstrated that implicit representations trained only for
reconstruction tasks typically generate encodings that are not useful for
semantic tasks. In this work, we propose a method that contextualises the
encodings of implicit representations, enabling their use in downstream tasks
(e.g. semantic segmentation), without requiring access to the original training
data or encoding network. Using an implicit representation trained for a
reconstruction task alone, our contextualising module takes an encoding trained
for reconstruction only and reveals meaningful semantic information that is
hidden in the encodings, without compromising the reconstruction performance.
With our proposed module, it becomes possible to pre-train implicit
representations on larger datasets, improving their reconstruction performance
compared to training on only a smaller labelled dataset, whilst maintaining
their segmentation performance on the labelled dataset. Importantly, our method
allows for future foundation implicit representation models to be fine-tuned on
unseen tasks, regardless of encoder or dataset availability.

本文提出一种语境化的方法，可以在不需要访问原始训练数据或编码网络的情况下，将专门用于重构的隐式表示的编码用于语义任务，揭示了编码中隐藏的有意义的语义信息，并通过在大型数据集上预训练隐式表示，同时保持在标签数据集上的附加分割和重构的性能，并允许对未见过任务进行微调。