We introduce CARETS, a systematic test suite to measure consistency and
robustness of modern VQA models through a series of six fine-grained capability
tests. In contrast to existing VQA test sets, CARETS features balanced question
generation to create pairs of instances to test models, with each pair focusing
on a specific capability such as rephrasing, logical symmetry or image
obfuscation. We evaluate six modern VQA systems on CARETS and identify several
actionable weaknesses in model comprehension, especially with concepts such as
negation, disjunction, or hypernym invariance. Interestingly, even the most
sophisticated models are sensitive to aspects such as swapping the order of
terms in a conjunction or varying the number of answer choices mentioned in the
question. We release CARETS to be used as an extensible tool for evaluating
multi-modal model robustness.

引入 CARETS 系统测试套件，通过一系列六种细粒度能力测试来衡量现代 VQA 模型的一致性和鲁棒性。我们在 CARETS 上评估了六个现代 VQA 系统，并确定了模型理解中的若干可操作性弱点，特别是在否定、或运算或超义不变性等概念方面。我们发布 CARETS 以作为可扩展的工具，用于评估多模态模型的鲁棒性。