Reconstructing natural videos from fMRI brain recordings is very challenging,
for two main reasons: (i) As fMRI data acquisition is difficult, we only have a
limited amount of supervised samples, which is not enough to cover the huge
space of natural videos; and (ii) The temporal resolution of fMRI recordings is
much lower than the frame rate of natural videos. In this paper, we propose a
self-supervised approach for natural-movie reconstruction. By employing
cycle-consistency over Encoding-Decoding natural videos, we can: (i) exploit
the full framerate of the training videos, and not be limited only to clips
that correspond to fMRI recordings; (ii) exploit massive amounts of external
natural videos which the subjects never saw inside the fMRI machine. These
enable increasing the applicable training data by several orders of magnitude,
introducing natural video priors to the decoding network, as well as temporal
coherence. Our approach significantly outperforms competing methods, since
those train only on the limited supervised data. We further introduce a new and
simple temporal prior of natural videos, which - when folded into our fMRI
decoder further - allows us to reconstruct videos at a higher frame-rate (HFR)
of up to x8 of the original fMRI sample rate.

本文研究自我监督方法来重建 fMRI 记录的自然视频，通过编码和解码自然视频的循环一致性以及利用大量外部自然视频，在提高适用的训练数据、引入视频先验以及保持时序一致性等方面取得了重大突破，相比于传统的只依靠有限受监督数据的方法，本文所提出的方法在性能上有了显著的提升。同时，作者还提出了一种简单的针对自然视频的时间先验，进一步提高了视频帧率。