Recently, intermediate feature maps of pre-trained convolutional neural
networks have shown significant perceptual quality improvements, when they are
used in the loss function for training new networks. It is believed that these
features are better at encoding the perceptual quality and provide more
efficient representations of input images compared to other perceptual metrics
such as SSIM and PSNR. However, there have been no systematic studies to
determine the underlying reason. Due to the lack of such an analysis, it is not
possible to evaluate the performance of a particular set of features or to
improve the perceptual quality even more by carefully selecting a subset of
features from a pre-trained CNN. This work shows that the capabilities of
pre-trained deep CNN features in optimizing the perceptual quality are
correlated with their success in capturing basic human visual perception
characteristics. In particular, we focus our analysis on fundamental aspects of
human perception, such as the contrast sensitivity and orientation selectivity.
We introduce two new formulations to measure the frequency and orientation
selectivity of the features learned by convolutional layers for evaluating deep
features learned by widely-used deep CNNs such as VGG-16. We demonstrate that
the pre-trained CNN features which receive higher scores are better at
predicting human quality judgment. Furthermore, we show the possibility of
using our method to select deep features to form a new loss function, which
improves the image reconstruction quality for the well-known single-image
super-resolution problem.

该研究分析了预训练卷积神经网络中的中间特征图如何提高感知质量，并提出了两种新的公式来评估深度 CNN 学习到的特征，发现使用这些特征能更好地预测人类质量判断， 进一步展示了利用该方法来选择深度特征用于形成新的损失函数可以提高单图超分辨率问题的图像重构质量。

深度表示为什么是良好的感知质量特征？

Why Are Deep Representations Good Perceptual Quality Features?

Classical models describe primary visual cortex (V1) as a filter bank of
orientation-selective linear-nonlinear (LN) or energy models, but these models
fail to predict neural responses to natural stimuli accurately. Recent work
shows that models based on convolutional neural networks (CNNs) lead to much
more accurate predictions, but it remains unclear which features are extracted
by V1 neurons beyond orientation selectivity and phase invariance. Here we work
towards systematically studying V1 computations by categorizing neurons into
groups that perform similar computations. We present a framework to identify
common features independent of individual neurons' orientation selectivity by
using a rotation-equivariant convolutional neural network, which automatically
extracts every feature at multiple different orientations. We fit this model to
responses of a population of 6000 neurons to natural images recorded in mouse
primary visual cortex using two-photon imaging. We show that our
rotation-equivariant network not only outperforms a regular CNN with the same
number of feature maps, but also reveals a number of common features shared by
many V1 neurons, which deviate from the typical textbook idea of V1 as a bank
of Gabor filters. Our findings are a first step towards a powerful new tool to
study the nonlinear computations in V1.

本文研究了鼠视觉皮层神经元的处理方式，通过使用旋转等价卷积神经网络提取共性特征并与传统模型进行比较，发现传统线性 - 非线性模型存在不足，但卷积神经网络能够更加准确地预测自然刺激下神经元的反应，且 V1 神经元存在一些与传统 Gabor 过滤器不同的共性特征，这对研究 V1 中的非线性计算具有重要意义。

初级视皮层旋转等变卷积神经网络模型

A rotation-equivariant convolutional neural network model of primary  visual cortex

Orientation selectivity is the most striking feature of simple cell coding in
V1 which has been shown to emerge from the reduction of higher-order
correlations in natural images in a large variety of statistical image models.
The most parsimonious one among these models is linear Independent Component
Analysis (ICA), whereas second-order decorrelation transformations such as
Principal Component Analysis (PCA) do not yield oriented filters. Because of
this finding it has been suggested that the emergence of orientation
selectivity may be explained by higher-order redundancy reduction. In order to
assess the tenability of this hypothesis, it is an important empirical question
how much more redundancies can be removed with ICA in comparison to PCA, or
other second-order decorrelation methods. This question has not yet been
settled, as over the last ten years contradicting results have been reported
ranging from less than five to more than hundred percent extra gain for ICA.
Here, we aim at resolving this conflict by presenting a very careful and
comprehensive analysis using three evaluation criteria related to redundancy
reduction: In addition to the multi-information and the average log-loss we
compute, for the first time, complete rate-distortion curves for ICA in
comparison with PCA. Without exception, we find that the advantage of the ICA
filters is surprisingly small. Furthermore, we show that a simple spherically
symmetric distribution with only two parameters can fit the data even better
than the probabilistic model underlying ICA. Since spherically symmetric models
are agnostic with respect to the specific filter shapes, we conlude that
orientation selectivity is unlikely to play a critical role for redundancy
reduction.

研究表明，在视觉皮层 V1 中，方向选择性源于自然图像中高阶冗余关联的降低，而线性独立成分分析模型 (ICA) 能够比主成分分析 (PCA) 更好地去除这种冗余，但我们发现 ICA 滤波器的优势非常小，并且简单的球对称分布可以比 ICA 更好地解释这一现象。因此，我们推断方向选择性对于冗余的减少不太可能起到关键作用。