Given a command, humans can directly execute the action after thinking or
choose to reject it, with reasonable feedback at the same time. However, the
behavior of existing text-to-image generation methods are uncontrollable and
irresponsible. In this paper, we construct extensive experiments to verify
whether they can be accountable (say no and explain why) for those prohibited
instructions. To this end, we define a novel text-based visual re-creation task
and construct new synthetic CLEVR-NOT dataset (620K) and manually pictured
Fruit-NOT dataset (50K). In our method, one text-image pair as the query is fed
into the machine, and the model gives a yes or no answer after visual and
textual reasoning. If the answer is yes, the image auto-encoder and
auto-regressive transformer must complete the visual re-creation under the
premise of ensuring image quality, otherwise the system needs to explain why
the commands cannot be completed or prohibited. We provide a detailed analysis
of experimental results in image quality, answer accuracy, and model behavior
in the face of uncertainty and imperfect user queries. Our results demonstrate
the difficulty of a single model for both textual and visual reasoning. We also
hope our explorations and findings can bring valuable insights about the
accountability of text-based image generation models. Code and datasets can be
found at this https URL.

本研究构建了一种基于文本和视觉推理的控制生成图像的方法，其中通过新构建的 CLEVR-NOT 数据集和手动制作的 Fruit-NOT 数据集对模型的准确性和行为进行了详细分析，旨在提高文本图像生成模型的可控性和可解释性。