The release of large datasets and developments in AI have led to dramatic
improvements in decoding methods that reconstruct seen images from human brain
activity. We evaluate the prospect of further improving recent decoding methods
by optimizing for consistency between reconstructions and brain activity during
inference. We sample seed reconstructions from a base decoding method, then
iteratively refine these reconstructions using a brain-optimized encoding model
that maps images to brain activity. At each iteration, we sample a small
library of images from an image distribution (a diffusion model) conditioned on
a seed reconstruction from the previous iteration. We select those that best
approximate the measured brain activity when passed through our encoding model,
and use these images for structural guidance during the generation of the small
library in the next iteration. We reduce the stochasticity of the image
distribution at each iteration, and stop when a criterion on the "width" of the
image distribution is met. We show that when this process is applied to recent
decoding methods, it outperforms the base decoding method as measured by human
raters, a variety of image feature metrics, and alignment to brain activity.
These results demonstrate that reconstruction quality can be significantly
improved by explicitly aligning decoding distributions to brain activity
distributions, even when the seed reconstruction is output from a
state-of-the-art decoding algorithm. Interestingly, the rate of refinement
varies systematically across visual cortex, with earlier visual areas generally
converging more slowly and preferring narrower image distributions, relative to
higher-level brain areas. Brain-optimized inference thus offers a succinct and
novel method for improving reconstructions and exploring the diversity of
representations across visual brain areas.

通过优化解码方法，将重构结果与脑活动一致性结合，通过迭代优化小样本图像库，我们证明了脑优化推理在改善重构质量和探索视觉脑区不同表征多样性方面的潜力。

脑优化推理改进功能磁共振成像的脑活动重建

Brain-optimized inference improves reconstructions of fMRI brain  activity

In this paper, we propose an adaptation to the area under the curve (AUC)
metric to measure the adversarial robustness of a model over a particular
$\epsilon$-interval $[\epsilon_0, \epsilon_1]$ (interval of adversarial
perturbation strengths) that facilitates unbiased comparisons across models
when they have different initial $\epsilon_0$ performance. This can be used to
determine how adversarially robust a model is to different image distributions
or task (or some other variable); and/or to measure how robust a model is
comparatively to other models. We used this adversarial robustness metric on
models of an MNIST, CIFAR-10, and a Fusion dataset (CIFAR-10 + MNIST) where
trained models performed either a digit or object recognition task using a
LeNet, ResNet50, or a fully connected network (FullyConnectedNet) architecture
and found the following: 1) CIFAR-10 models are inherently less adversarially
robust than MNIST models; 2) Both the image distribution and task that a model
is trained on can affect the adversarial robustness of the resultant model. 3)
Pretraining with a different image distribution and task sometimes carries over
the adversarial robustness induced by that image distribution and task in the
resultant model; Collectively, our results imply non-trivial differences of the
learned representation space of one perceptual system over another given its
exposure to different image statistics or tasks (mainly objects vs digits).
Moreover, these results hold even when model systems are equalized to have the
same level of performance, or when exposed to approximately matched image
statistics of fusion images but with different tasks.

该研究提出了一种新的指标来度量模型的鲁棒性，能够进行跨模型的比较，并在实验中发现训练数据集以及任务类型的差异影响模型的鲁棒性。

图像分布和任务对敌对鲁棒性的影响

The Effects of Image Distribution and Task on Adversarial Robustness

Probability density estimation is a classical and well studied problem, but
standard density estimation methods have historically lacked the power to model
complex and high-dimensional image distributions. More recent generative models
leverage the power of neural networks to implicitly learn and represent
probability models over complex images. We describe methods to extract explicit
probability density estimates from GANs, and explore the properties of these
image density functions. We perform sanity check experiments to provide
evidence that these probabilities are reasonable. However, we also show that
density functions of natural images are difficult to interpret and thus limited
in use. We study reasons for this lack of interpretability, and show that we
can get interpretability back by doing density estimation on latent
representations of images.

本文研究利用生成对抗网络（GAN）从复杂高维图像中提取明确的概率密度估计方法，发现对自然图像进行密度函数估计难以解释，但是在图像的潜在表示上进行密度估计可以获得可解释性。

利用生成模型理解自然图像分布的（不）可解释性

Understanding the (un)interpretability of natural image distributions  using generative models

CycleGAN (Zhu et al. 2017) is one recent successful approach to learn a
transformation between two image distributions. In a series of experiments, we
demonstrate an intriguing property of the model: CycleGAN learns to "hide"
information about a source image into the images it generates in a nearly
imperceptible, high-frequency signal. This trick ensures that the generator can
recover the original sample and thus satisfy the cyclic consistency
requirement, while the generated image remains realistic. We connect this
phenomenon with adversarial attacks by viewing CycleGAN's training procedure as
training a generator of adversarial examples and demonstrate that the cyclic
consistency loss causes CycleGAN to be especially vulnerable to adversarial
attacks.

CycleGAN 可以学习到将两个图像分布之间的转化，并且会将源图像信息隐藏在其生成的图像中，从而保证了循环一致性，然而其通过循环一致性来保证生成的图像真实性导致了其容易受到对抗攻击的影响。