The release of large datasets and developments in AI have led to dramatic
improvements in decoding methods that reconstruct seen images from human brain
activity. We evaluate the prospect of further improving recent decoding methods
by optimizing for consistency between reconstructions and brain activity during
inference. We sample seed reconstructions from a base decoding method, then
iteratively refine these reconstructions using a brain-optimized encoding model
that maps images to brain activity. At each iteration, we sample a small
library of images from an image distribution (a diffusion model) conditioned on
a seed reconstruction from the previous iteration. We select those that best
approximate the measured brain activity when passed through our encoding model,
and use these images for structural guidance during the generation of the small
library in the next iteration. We reduce the stochasticity of the image
distribution at each iteration, and stop when a criterion on the "width" of the
image distribution is met. We show that when this process is applied to recent
decoding methods, it outperforms the base decoding method as measured by human
raters, a variety of image feature metrics, and alignment to brain activity.
These results demonstrate that reconstruction quality can be significantly
improved by explicitly aligning decoding distributions to brain activity
distributions, even when the seed reconstruction is output from a
state-of-the-art decoding algorithm. Interestingly, the rate of refinement
varies systematically across visual cortex, with earlier visual areas generally
converging more slowly and preferring narrower image distributions, relative to
higher-level brain areas. Brain-optimized inference thus offers a succinct and
novel method for improving reconstructions and exploring the diversity of
representations across visual brain areas.

通过优化解码方法，将重构结果与脑活动一致性结合，通过迭代优化小样本图像库，我们证明了脑优化推理在改善重构质量和探索视觉脑区不同表征多样性方面的潜力。