The challenge of open-vocabulary recognition lies in the model has no clue of
new categories it is applied to. Existing works have proposed different methods
to embed category cues into the model, \eg, through few-shot fine-tuning,
providing category names or textual descriptions to Vision-Language Models.
Fine-tuning is time-consuming and degrades the generalization capability.
Textual descriptions could be ambiguous and fail to depict visual details. This
paper tackles open-vocabulary recognition from a different perspective by
referring to multi-modal clues composed of textual descriptions and exemplar
images. Our method, named OVMR, adopts two innovative components to pursue a
more robust category cues embedding. A multi-modal classifier is first
generated by dynamically complementing textual descriptions with image
exemplars. A preference-based refinement module is hence applied to fuse
uni-modal and multi-modal classifiers, with the aim to alleviate issues of
low-quality exemplar images or textual descriptions. The proposed OVMR is a
plug-and-play module, and works well with exemplar images randomly crawled from
the Internet. Extensive experiments have demonstrated the promising performance
of OVMR, \eg, it outperforms existing methods across various scenarios and
setups. Codes are publicly available at
\href{https://github.com/Zehong-Ma/OVMR}{this https URL}.

本文通过引用由文本描述和示例图像组成的多模式线索，以不同的角度解决开放词汇识别问题，并提出了一种名为 OVMR 的方法，该方法采用两个创新组件来追求更强的类别线索嵌入。

OVMR: 多模式参考下的开放词汇识别

OVMR: Open-Vocabulary Recognition with Multi-Modal References

This study introduces Text-Guided Subject-Driven Image Inpainting, a novel
task that combines text and exemplar images for image inpainting. While both
text and exemplar images have been used independently in previous efforts,
their combined utilization remains unexplored. Simultaneously accommodating
both conditions poses a significant challenge due to the inherent balance
required between editability and subject fidelity. To tackle this challenge, we
propose a two-step approach DreamInpainter. First, we compute dense subject
features to ensure accurate subject replication. Then, we employ a
discriminative token selection module to eliminate redundant subject details,
preserving the subject's identity while allowing changes according to other
conditions such as mask shape and text prompts. Additionally, we introduce a
decoupling regularization technique to enhance text control in the presence of
exemplar images. Our extensive experiments demonstrate the superior performance
of our method in terms of visual quality, identity preservation, and text
control, showcasing its effectiveness in the context of text-guided
subject-driven image inpainting.

本研究介绍了一种新颖的任务 —— 基于文本导向的主题驱动图像修复，它将文本和示例图像结合起来进行图像修复。通过同时适应这两个条件，我们提出了一种两步法的方法 ——DreamInpainter，该方法通过计算密集的主题特征来确保准确的主题复制，并利用一个有区分性的令牌选择模块来消除多余的主题细节，保留主题的身份，同时允许根据其他条件（如遮罩形状和文本提示）进行更改。此外，我们引入了一种解耦正则化技术，以增强在存在示例图像的情况下对文本的控制。我们广泛的实验证明了我们的方法在视觉质量、身份保护和文本控制方面的卓越性能，展示了它在文本导向的主题驱动图像修复的情境中的有效性。

DreamInpainter: 基于文本引导的主题驱动图像修复通过扩散模型

DreamInpainter: Text-Guided Subject-Driven Image Inpainting with  Diffusion Models

Object Transfiguration replaces an object in an image with another object
from a second image. For example it can perform tasks like "putting exactly
those eyeglasses from image A on the nose of the person in image B". Usage of
exemplar images allows more precise specification of desired modifications and
improves the diversity of conditional image generation. However, previous
methods that rely on feature space operations, require paired data and/or
appearance models for training or disentangling objects from background. In
this work, we propose a model that can learn object transfiguration from two
unpaired sets of images: one set containing images that "have" that kind of
object, and the other set being the opposite, with the mild constraint that the
objects be located approximately at the same place. For example, the training
data can be one set of reference face images that have eyeglasses, and another
set of images that have not, both of which spatially aligned by face landmarks.
Despite the weak 0/1 labels, our model can learn an "eyeglasses" subspace that
contain multiple representatives of different types of glasses. Consequently,
we can perform fine-grained control of generated images, like swapping the
glasses in two images by swapping the projected components in the "eyeglasses"
subspace, to create novel images of people wearing eyeglasses.
Overall, our deterministic generative model learns disentangled attribute
subspaces from weakly labeled data by adversarial training. Experiments on
CelebA and Multi-PIE datasets validate the effectiveness of the proposed model
on real world data, in generating images with specified eyeglasses, smiling,
hair styles, and lighting conditions etc. The code is available online.

本文介绍了一种利用样例图像进行对象转换的模型，该模型可以从两组不配对的图像中学习对象转换，通过生成模型从弱标记的数据中学习分离的属性子空间。该模型在 CelebA 和 Multi-PIE 数据集上进行了实验验证。