A defining characteristic of natural vision is its ability to withstand a
variety of input alterations, resulting in the creation of an invariant
representation of the surroundings. While convolutional neural networks exhibit
resilience to certain forms of spatial input variation, modifications in the
spatial and temporal aspects can significantly affect the representations of
video content in deep neural networks. Inspired by the resilience of natural
vision to input variations, we employ a simple multi-stream model to explore
its potential to address spatiotemporal changes by including temporal features.
Our primary goal is to introduce a video-trained model and evaluate its
robustness to diverse image and video inputs, with a particular focus on
exploring the role of temporal features in invariant recognition. Results show
that including videos and the temporal stream during training mitigates the
decline in accuracy and mAP in image and video understanding tasks by 1.36% and
3.14%, respectively.

通过多流模型，我们引入一个经过视频训练的模型，并评估其在各种图像和视频输入下的鲁棒性，特别关注时间特征在不变识别中的作用。结果表明，在训练中包含视频和时间流能够抑制图像和视频理解任务中准确性和平均准确率（mAP）下降分别约 1.36％和 3.14％。

超越静态图像：稳健的多流时空网络

Beyond Still Images: Robust Multi-Stream Spatiotemporal Networks

Convolutional neural network (CNN) driven by image recognition has been shown
to be able to explain cortical responses to static pictures at ventral-stream
areas. Here, we further showed that such CNN could reliably predict and decode
functional magnetic resonance imaging data from humans watching natural movies,
despite its lack of any mechanism to account for temporal dynamics or feedback
processing. Using separate data, encoding and decoding models were developed
and evaluated for describing the bi-directional relationships be-tween the CNN
and the brain. Through the encoding models, the CNN-predicted areas covered not
only the ventral stream, but also the dorsal stream, albe-it to a lesser
degree; single-voxel response was visualized as the specific pixel pattern that
drove the response, revealing the distinct representation of individual
cortical location; cortical activation was synthesized from natural images with
high-throughput to map category representation, con-trast, and selectivity.
Through the decoding models, fMRI signals were directly decoded to estimate the
feature representations in both visual and semantic spaces, for direct visual
reconstruction and seman-tic categorization, respectively. These results
cor-roborate, generalize, and extend previous findings, and highlight the value
of using deep learning, as an all-in-one model of the visual cortex, to
understand and decode natural vision.

研究表明，使用卷积神经网络 (CNN) 驱动的图像识别技术不仅能在下流视觉区域说明大脑对静态图片的反应，还能可靠地预测和解码人类观看自然电影时的功能性磁共振成像数据，并通过缺少有关时间动态或反馈处理机制的 CNN-Predicted areas 覆盖了包括腹侧流和背侧流在内的大部分视觉和语义空间。