Most Visual Question Answering (VQA) models suffer from the language prior
problem, which is caused by inherent data biases. Specifically, VQA models tend
to answer questions (e.g., what color is the banana?) based on the
high-frequency answers (e.g., yellow) ignoring image contents. Existing
approaches tackle this problem by creating delicate models or introducing
additional visual annotations to reduce question dependency while strengthening
image dependency. However, they are still subject to the language prior problem
since the data biases have not been even alleviated. In this paper, we
introduce a self-supervised learning framework to solve this problem.
Concretely, we first automatically generate labeled data to balance the biased
data, and propose a self-supervised auxiliary task to utilize the balanced data
to assist the base VQA model to overcome language priors. Our method can
compensate for the data biases by generating balanced data without introducing
external annotations. Experimental results show that our method can
significantly outperform the state-of-the-art, improving the overall accuracy
from 49.50% to 57.59% on the most commonly used benchmark VQA-CP v2. In other
words, we can increase the performance of annotation-based methods by 16%
without using external annotations.

本研究介绍了一种自监督学习框架来解决视觉问答模型中固有数据偏差问题。我们利用自动生成的标签数据来平衡数据偏差，提出一种自监督辅助任务来减少语言先验的影响，从而提高 VQA 模型的准确性，无需引入额外的可视化注释。实验结果表明，我们的方法在常用基准 VQA-CP v2 上将整体准确性从 49.50％提高到 57.59％，可以将基于注释的方法的性能提高 16％，而无需使用外部注释。

通过自监督学习克服语言先验的视觉问答

Overcoming Language Priors with Self-supervised Learning for Visual  Question Answering

Benefiting from the advancement of computer vision, natural language
processing and information retrieval techniques, visual question answering
(VQA), which aims to answer questions about an image or a video, has received
lots of attentions over the past few years. Although some progress has been
achieved so far, several studies have pointed out that current VQA models are
heavily affected by the \emph{language prior problem}, which means they tend to
answer questions based on the co-occurrence patterns of question keywords
(e.g., how many) and answers (e.g., 2) instead of understanding images and
questions. Existing methods attempt to solve this problem by either balancing
the biased datasets or forcing models to better understand images. However,
only marginal effects and even performance deterioration are observed for the
first and second solution, respectively. In addition, another important issue
is the lack of measurement to quantitatively measure the extent of the language
prior effect, which severely hinders the advancement of related techniques.
In this paper, we make contributions to solve the above problems from two
perspectives. Firstly, we design a metric to quantitatively measure the
language prior effect of VQA models. The proposed metric has been demonstrated
to be effective in our empirical studies. Secondly, we propose a regularization
method (i.e., score regularization module) to enhance current VQA models by
alleviating the language prior problem as well as boosting the backbone model
performance. The proposed score regularization module adopts a pair-wise
learning strategy, which makes the VQA models answer the question based on the
reasoning of the image (upon this question) instead of basing on
question-answer patterns observed in the biased training set. The score
regularization module is flexible to be integrated into various VQA models.

本文针对视觉问答技术所面临的语言先验问题，设计了一个度量指标并提出了一种得分正则化模块，该模块采用成对学习方法，可以缓解语言先验问题，并提高骨干模型的性能。