Understanding how visual information is encoded in biological and artificial
systems often requires vision scientists to generate appropriate stimuli to
test specific hypotheses. Although deep neural network models have
revolutionized the field of image generation with methods such as image style
transfer, available methods for video generation are scarce. Here, we introduce
the Spatiotemporal Style Transfer (STST) algorithm, a dynamic visual stimulus
generation framework that allows powerful manipulation and synthesis of video
stimuli for vision research. It is based on a two-stream deep neural network
model that factorizes spatial and temporal features to generate dynamic visual
stimuli whose model layer activations are matched to those of input videos. As
an example, we show that our algorithm enables the generation of model
metamers, dynamic stimuli whose layer activations within our two-stream model
are matched to those of natural videos. We show that these generated stimuli
match the low-level spatiotemporal features of their natural counterparts but
lack their high-level semantic features, making it a powerful paradigm to study
object recognition. Late layer activations in deep vision models exhibited a
lower similarity between natural and metameric stimuli compared to early
layers, confirming the lack of high-level information in the generated stimuli.
Finally, we use our generated stimuli to probe the representational
capabilities of predictive coding deep networks. These results showcase
potential applications of our algorithm as a versatile tool for dynamic
stimulus generation in vision science.

图像研究中，我们引入了动态视觉刺激生成框架的 Spaciotemporal Style Transfer 算法，它基于双流深度神经网络模型，能够生成与自然视频的低级时空特征匹配的模型变形器，缺乏高级语义特征，成为研究对象识别的有力范例。