Machine learning advances in the last decade have relied significantly on
large-scale datasets that continue to grow in size. Increasingly, those
datasets also contain different data modalities. However, large multi-modal
datasets are hard to annotate, and annotations may contain biases that we are
often unaware of. Deep-net-based classifiers, in turn, are prone to exploit
those biases and to find shortcuts. To study and quantify this concern, we
introduce the perceptual score, a metric that assesses the degree to which a
model relies on the different subsets of the input features, i.e., modalities.
Using the perceptual score, we find a surprisingly consistent trend across four
popular datasets: recent, more accurate state-of-the-art multi-modal models for
visual question-answering or visual dialog tend to perceive the visual data
less than their predecessors. This trend is concerning as answers are hence
increasingly inferred from textual cues only. Using the perceptual score also
helps to analyze model biases by decomposing the score into data subset
contributions. We hope to spur a discussion on the perceptiveness of
multi-modal models and also hope to encourage the community working on
multi-modal classifiers to start quantifying perceptiveness via the proposed
perceptual score.

本文提出了感知分数的概念，通过该指标对多模态数据集中的各种输入特征进行评估，发现最新的视觉问答或视觉对话多模态模型相较于它们的先驱越来越少重视视觉数据，这趋势引起了担忧，需要对此展开讨论并提出解决思路。

感知得分：您的模型感知哪些数据模态？

Perceptual Score: What Data Modalities Does Your Model Perceive?

Inspired by the free-energy brain theory, which implies that human visual
system (HVS) tends to reduce uncertainty and restore perceptual details upon
seeing a distorted image, we propose restorative adversarial net (RAN), a
GAN-based model for no-reference image quality assessment (NR-IQA). RAN, which
mimics the process of HVS, consists of three components: a restorator, a
discriminator and an evaluator. The restorator restores and reconstructs input
distorted image patches, while the discriminator distinguishes the
reconstructed patches from the pristine distortion-free patches. After
restoration, we observe that the perceptual distance between the restored and
the distorted patches is monotonic with respect to the distortion level. We
further define Gain of Restoration (GoR) based on this phenomenon. The
evaluator predicts perceptual score by extracting feature representations from
the distorted and restored patches to measure GoR. Eventually, the quality
score of an input image is estimated by weighted sum of the patch scores.
Experimental results on Waterloo Exploration, LIVE and TID2013 show the
effectiveness and generalization ability of RAN compared to the
state-of-the-art NR-IQA models.

通过模拟人类视觉系统的过程，我们提出了一种无参考图像质量评估系统 ——Restorative Adversarial Net，该系统包括复原器、鉴别器和评估器三个部分，通过提取特征来计算 Gain of Restoration 并将其结合用于评估输入图片的质量，实验证明该系统相比其他先进的 NR-IQA 模型具有更高的效率和泛化能力。