In a surge of text-to-image (T2I) models and their customization methods that
generate new images of a user-provided subject, current works focus on
alleviating the costs incurred by a lengthy per-subject optimization. These
zero-shot customization methods encode the image of a specified subject into a
visual embedding which is then utilized alongside the textual embedding for
diffusion guidance. The visual embedding incorporates intrinsic information
about the subject, while the textual embedding provides a new, transient
context. However, the existing methods often 1) are significantly affected by
the input images, eg., generating images with the same pose, and 2) exhibit
deterioration in the subject's identity. We first pin down the problem and show
that redundant pose information in the visual embedding interferes with the
textual embedding containing the desired pose information. To address this
issue, we propose orthogonal visual embedding which effectively harmonizes with
the given textual embedding. We also adopt the visual-only embedding and inject
the subject's clear features utilizing a self-attention swap. Our results
demonstrate the effectiveness and robustness of our method, which offers highly
flexible zero-shot generation while effectively maintaining the subject's
identity.

通过引入正交视觉嵌入并利用自我注意力交换，我们提出了一种高效维护主题身份的方法，以解决文本到图像模型中遭遇的问题，并展示了该方法的效果和鲁棒性。

视觉和文本嵌入的零样本文本到图像定制的协调

Harmonizing Visual and Textual Embeddings for Zero-Shot Text-to-Image  Customization

Recent advancements in deep learning have led to the development of powerful
language models (LMs) that excel in various tasks. Despite these achievements,
there is still room for improvement, particularly in enhancing reasoning
abilities and incorporating multimodal data. This report investigates the
potential impact of combining Chain-of-Thought (CoT) reasoning and Visual
Question Answering (VQA) techniques to improve LM's accuracy in solving
multiple-choice questions. By employing TextVQA and ScienceQA datasets, we
assessed the effectiveness of three text embedding methods and three visual
embedding approaches. Our experiments aimed to fill the gap in current research
by investigating the combined impact of CoT and VQA, contributing to the
understanding of how these techniques can improve the reasoning capabilities of
state-of-the-art models like GPT-4. Results from our experiments demonstrated
the potential of these approaches in enhancing LM's reasoning and
question-answering capabilities, providing insights for further research and
development in the field, and paving the way for more accurate and reliable AI
systems that can handle complex reasoning tasks across multiple modalities.

利用链式思维和视觉问答技术，通过评估文本嵌入方法和视觉嵌入方法的有效性，研究如何提高深度学习模型在解决多项选择问题方面的准确性，实验结果显示这些方法在增强推理和问答能力方面具有潜力。

实现统一的多模态推理框架

Towards a Unified Multimodal Reasoning Framework

We propose a novel Generalized Zero-Shot learning (GZSL) method that is
agnostic to both unseen images and unseen semantic vectors during training.
Prior works in this context propose to map high-dimensional visual features to
the semantic domain, we believe contributes to the semantic gap. To bridge the
gap, we propose a novel low-dimensional embedding of visual instances that is
"visually semantic." Analogous to semantic data that quantifies the existence
of an attribute in the presented instance, components of our visual embedding
quantifies existence of a prototypical part-type in the presented instance. In
parallel, as a thought experiment, we quantify the impact of noisy semantic
data by utilizing a novel visual oracle to visually supervise a learner. These
factors, namely semantic noise, visual-semantic gap and label noise lead us to
propose a new graphical model for inference with pairwise interactions between
label, semantic data, and inputs. We tabulate results on a number of benchmark
datasets demonstrating significant improvement in accuracy over
state-of-the-art under both semantic and visual supervision.

本文提出了一种新颖的广义零样本学习（GZSL）方法，它对训练期间的未见图像和未见语义向量具有不可知性。通过提出一种视觉实例的低维嵌入来打破视觉 - 语义间隙，并借助一个新的视觉神谕来量化噪声语义数据的影响，以提高准确性。在一系列数据集上用图模型进行推理的实验结果显示，该方法在语义和视觉监督下均明显优于现有技术。