Humans experience the world through constantly changing visual stimuli, where
scenes can shift and move, change in appearance, and vary in distance. The
dynamic nature of visual perception is a fundamental aspect of our daily lives,
yet the large majority of research on object and scene processing, particularly
using fMRI, has focused on static stimuli. While studies of static image
perception are attractive due to their computational simplicity, they impose a
strong non-naturalistic constraint on our investigation of human vision. In
contrast, dynamic visual stimuli offer a more ecologically-valid approach but
present new challenges due to the interplay between spatial and temporal
information, making it difficult to disentangle the representations of stable
image features and motion. To overcome this limitation -- given dynamic inputs,
we explicitly decouple the modeling of static image representations and motion
representations in the human brain. Three results demonstrate the feasibility
of this approach. First, we show that visual motion information as optical flow
can be predicted (or decoded) from brain activity as measured by fMRI. Second,
we show that this predicted motion can be used to realistically animate static
images using a motion-conditioned video diffusion model (where the motion is
driven by fMRI brain activity). Third, we show prediction in the reverse
direction: existing video encoders can be fine-tuned to predict fMRI brain
activity from video imagery, and can do so more effectively than image
encoders. This foundational work offers a novel, extensible framework for
interpreting how the human brain processes dynamic visual information.

人类通过不断变化的视觉刺激来感知世界，其中场景可以移动、改变外观和距离。这篇研究论文针对动态视觉刺激的大多数研究聚焦于静态刺激，限制了对人类视觉的调查。与之相反，动态视觉刺激提供了更符合生态有效性的方法，但由于时空信息的相互作用，使得解耦稳定图像特征和运动表征的困难增加。为了解决这个问题，在动态输入情况下，我们明确地解耦了人脑中静态图像表征和运动表征的建模。三个结果证明了这个方法的可行性。首先，我们展示了视觉运动信息（如光流）可以从 fMRI 测得的脑活动中预测（或解码）出来。其次，我们展示了可以使用这个预测运动来使用基于运动条件的视频扩散模型（其中运动由 fMRI 脑活动驱动）逼真地使静态图像动画化。第三，我们展示了在相反方向上的预测：现有的视频编码器可以进行微调，从视频图像中预测 fMRI 脑活动，并且相较于图像编码器能更有效地进行预测。这项基础工作为解释人类大脑如何处理动态视觉信息提供了一种新颖且可扩展的框架。

动态视觉刺激的神经表征

Neural Representations of Dynamic Visual Stimuli

Understanding how visual information is encoded in biological and artificial
systems often requires vision scientists to generate appropriate stimuli to
test specific hypotheses. Although deep neural network models have
revolutionized the field of image generation with methods such as image style
transfer, available methods for video generation are scarce. Here, we introduce
the Spatiotemporal Style Transfer (STST) algorithm, a dynamic visual stimulus
generation framework that allows powerful manipulation and synthesis of video
stimuli for vision research. It is based on a two-stream deep neural network
model that factorizes spatial and temporal features to generate dynamic visual
stimuli whose model layer activations are matched to those of input videos. As
an example, we show that our algorithm enables the generation of model
metamers, dynamic stimuli whose layer activations within our two-stream model
are matched to those of natural videos. We show that these generated stimuli
match the low-level spatiotemporal features of their natural counterparts but
lack their high-level semantic features, making it a powerful paradigm to study
object recognition. Late layer activations in deep vision models exhibited a
lower similarity between natural and metameric stimuli compared to early
layers, confirming the lack of high-level information in the generated stimuli.
Finally, we use our generated stimuli to probe the representational
capabilities of predictive coding deep networks. These results showcase
potential applications of our algorithm as a versatile tool for dynamic
stimulus generation in vision science.

图像研究中，我们引入了动态视觉刺激生成框架的 Spaciotemporal Style Transfer 算法，它基于双流深度神经网络模型，能够生成与自然视频的低级时空特征匹配的模型变形器，缺乏高级语义特征，成为研究对象识别的有力范例。