Visual question answering (VQA) is a fundamental and essential AI task, and
VQA-based disaster scenario understanding is a hot research topic. For
instance, we can ask questions about a disaster image by the VQA model and the
answer can help identify whether anyone or anything is affected by the
disaster. However, previous VQA models for disaster damage assessment have some
shortcomings, such as limited candidate answer space, monotonous question
types, and limited answering capability of existing models. In this paper, we
propose a zero-shot VQA model named Zero-shot VQA for Flood Disaster Damage
Assessment (ZFDDA). It is a VQA model for damage assessment without
pre-training. Also, with flood disaster as the main research object, we build a
Freestyle Flood Disaster Image Question Answering dataset (FFD-IQA) to evaluate
our VQA model. This new dataset expands the question types to include
free-form, multiple-choice, and yes-no questions. At the same time, we expand
the size of the previous dataset to contain a total of 2,058 images and 22,422
question-meta ground truth pairs. Most importantly, our model uses
well-designed chain of thought (CoT) demonstrations to unlock the potential of
the large language model, allowing zero-shot VQA to show better performance in
disaster scenarios. The experimental results show that the accuracy in
answering complex questions is greatly improved with CoT prompts. Our study
provides a research basis for subsequent research of VQA for other disaster
scenarios.

基于大规模语言模型 CoT 链条提示的零样本 VQA 模型，通过构建 FFD-IQA 评估数据集，对洪水灾害进行损坏评估，并在复杂问题回答中展现了更好的性能。

释放大型语言模型的潜力：洪水灾害场景下的无监督视觉问答（Zero-shot VQA）

Unleashing the Potential of Large Language Model: Zero-shot VQA for  Flood Disaster Scenario

The field of text-conditioned image generation has made unparalleled progress
with the recent advent of latent diffusion models. While remarkable, as the
complexity of given text input increases, the state-of-the-art diffusion models
may still fail in generating images which accurately convey the semantics of
the given prompt. Furthermore, it has been observed that such misalignments are
often left undetected by pretrained multi-modal models such as CLIP. To address
these problems, in this paper we explore a simple yet effective decompositional
approach towards both evaluation and improvement of text-to-image alignment. In
particular, we first introduce a Decompositional-Alignment-Score which given a
complex prompt decomposes it into a set of disjoint assertions. The alignment
of each assertion with generated images is then measured using a VQA model.
Finally, alignment scores for different assertions are combined aposteriori to
give the final text-to-image alignment score. Experimental analysis reveals
that the proposed alignment metric shows significantly higher correlation with
human ratings as opposed to traditional CLIP, BLIP scores. Furthermore, we also
find that the assertion level alignment scores provide a useful feedback which
can then be used in a simple iterative procedure to gradually increase the
expression of different assertions in the final image outputs. Human user
studies indicate that the proposed approach surpasses previous state-of-the-art
by 8.7% in overall text-to-image alignment accuracy. Project page for our paper
is available at this https URL

通过分解式对齐评估和改进文本到图像的对齐效果，并使用 Decompositional-Alignment-Score 和 VQA 模型来测量不同断言的对齐度。实验结果表明，这种对齐度指标与人类评分高度相关，并且断言级别的对齐度评分可用于逐步提高最终图像输出中不同断言的表达。人类用户研究表明，该方法在整体文本到图像对齐准确性方面超过了之前的最先进方法 8.7%。

划分、评估与改进：基于迭代 VQA 反馈的文本到图像对齐的评估和改进

Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image  Alignment with Iterative VQA Feedback

While Visual Question Answering (VQA) models continue to push the
state-of-the-art forward, they largely remain black-boxes - failing to provide
insight into how or why an answer is generated. In this ongoing work, we
propose addressing this shortcoming by learning to generate counterfactual
images for a VQA model - i.e. given a question-image pair, we wish to generate
a new image such that i) the VQA model outputs a different answer, ii) the new
image is minimally different from the original, and iii) the new image is
realistic. Our hope is that providing such counterfactual examples allows users
to investigate and understand the VQA model's internal mechanisms.

通过学习生成对抗性图片对视觉问答模型进行解释，从而使得人们可以理解其内部机制。