Reconstructing natural videos from fMRI brain recordings is very challenging,
for two main reasons: (i) As fMRI data acquisition is difficult, we only have a
limited amount of supervised samples, which is not enough to cover the huge
space of natural videos; and (ii) The temporal resolution of fMRI recordings is
much lower than the frame rate of natural videos. In this paper, we propose a
self-supervised approach for natural-movie reconstruction. By employing
cycle-consistency over Encoding-Decoding natural videos, we can: (i) exploit
the full framerate of the training videos, and not be limited only to clips
that correspond to fMRI recordings; (ii) exploit massive amounts of external
natural videos which the subjects never saw inside the fMRI machine. These
enable increasing the applicable training data by several orders of magnitude,
introducing natural video priors to the decoding network, as well as temporal
coherence. Our approach significantly outperforms competing methods, since
those train only on the limited supervised data. We further introduce a new and
simple temporal prior of natural videos, which - when folded into our fMRI
decoder further - allows us to reconstruct videos at a higher frame-rate (HFR)
of up to x8 of the original fMRI sample rate.

本文研究自我监督方法来重建 fMRI 记录的自然视频，通过编码和解码自然视频的循环一致性以及利用大量外部自然视频，在提高适用的训练数据、引入视频先验以及保持时序一致性等方面取得了重大突破，相比于传统的只依靠有限受监督数据的方法，本文所提出的方法在性能上有了显著的提升。同时，作者还提出了一种简单的针对自然视频的时间先验，进一步提高了视频帧率。

一分钱之价（视觉）：自监督重建自然电影的脑活动

A Penny for Your (visual) Thoughts: Self-Supervised Reconstruction of Natural Movies from Brain Activity

Existing works address the problem of generating high frame-rate sharp videos
by separately learning the frame deblurring and frame interpolation modules.
Most of these approaches have a strong prior assumption that all the input
frames are blurry whereas in a real-world setting, the quality of frames
varies. Moreover, such approaches are trained to perform either of the two
tasks - deblurring or interpolation - in isolation, while many practical
situations call for both. Different from these works, we address a more
realistic problem of high frame-rate sharp video synthesis with no prior
assumption that input is always blurry. We introduce a novel architecture,
Adaptive Latent Attention Network (ALANET), which synthesizes sharp high
frame-rate videos with no prior knowledge of input frames being blurry or not,
thereby performing the task of both deblurring and interpolation. We
hypothesize that information from the latent representation of the consecutive
frames can be utilized to generate optimized representations for both frame
deblurring and frame interpolation. Specifically, we employ combination of
self-attention and cross-attention module between consecutive frames in the
latent space to generate optimized representation for each frame. The optimized
representation learnt using these attention modules help the model to generate
and interpolate sharp frames. Extensive experiments on standard datasets
demonstrate that our method performs favorably against various state-of-the-art
approaches, even though we tackle a much more difficult problem.

本文提出了一种新颖的 Adaptive Latent Attention Network (ALANET) 体系结构，可生成高帧率锐利视频，具有去模糊和插值任务，利用自注意和交叉注意模块优化表示来帮助模型生成和插值锐利帧。实验结果表明，我们的方法在标准数据集上表现优异。