Text to image generation methods (T2I) are widely popular in generating art
and other creative artifacts. While visual hallucinations can be a positive
factor in scenarios where creativity is appreciated, such artifacts are poorly
suited for cases where the generated image needs to be grounded in complex
natural language without explicit visual elements. In this paper, we propose to
strengthen the consistency property of T2I methods in the presence of natural
complex language, which often breaks the limits of T2I methods by including
non-visual information, and textual elements that require knowledge for
accurate generation. To address these phenomena, we propose a Natural Language
to Verified Image generation approach (NL2VI) that converts a natural prompt
into a visual prompt, which is more suitable for image generation. A T2I model
then generates an image for the visual prompt, which is then verified with VQA
algorithms. Experimentally, aligning natural prompts with image generation can
improve the consistency of the generated images by up to 11% over the state of
the art. Moreover, improvements can generalize to challenging domains like
cooking and DIY tasks, where the correctness of the generated image is crucial
to illustrate actions.

本研究提出一种自然语言到有效图像生成（NL2VI）的方法，将自然提示转换为更适合图像生成的可视提示。通过实验，将自然提示与图像生成对齐可以提高生成图像的一致性，尤其在烹饪和 DIY 等领域具有广泛的推广和应用。

从自然语言到可验证图像生成的视觉属性转移

Transferring Visual Attributes from Natural Language to Verified Image  Generation

Visual Question Answering (VQA) research is split into two camps: the first
focuses on VQA datasets that require natural image understanding and the second
focuses on synthetic datasets that test reasoning. A good VQA algorithm should
be capable of both, but only a few VQA algorithms are tested in this manner. We
compare five state-of-the-art VQA algorithms across eight VQA datasets covering
both domains. To make the comparison fair, all of the models are standardized
as much as possible, e.g., they use the same visual features, answer
vocabularies, etc. We find that methods do not generalize across the two
domains. To address this problem, we propose a new VQA algorithm that rivals or
exceeds the state-of-the-art for both domains.

该论文研究了视觉问答算法，发现现有算法在自然图像理解和推理两个领域上不能很好地综合运用。作者提出了一种新的 VQA 算法，能够在这两个领域上达到或超越现有算法的水平。