We consider the problem of video snapshot compressive imaging (SCI), where sequential high-speed frames are modulated by different masks and captured by a single measurement. The underlying principle of reconstructing multi-frame images from only one single measurement is to solve an ill-posed problem. By combining optimization algorithms and neural networks, deep unfolding networks (DUNs) score tremendous achievements in solving inverse problems. In this paper, our proposed model is under the DUN framework and we propose a 3D Convolution-Transformer Mixture (CTM) module with a 3D efficient and scalable attention model plugged in, which helps fully learn the correlation between temporal and spatial dimensions by virtue of Transformer. To our best knowledge, this is the first time that Transformer is employed to video SCI reconstruction. Besides, to further investigate the high-frequency information during the reconstruction process which are neglected in previous studies, we introduce variance estimation characterizing the uncertainty on a pixel-by-pixel basis. Extensive experimental results demonstrate that our proposed method achieves state-of-the-art (SOTA) (with a 1.2dB gain in PSNR over previous SOTA algorithm) results. We will release the code.

本文提出了一种基于DUN框架下的3D卷积-Transformer混合（CTM）模块，该模块利用Transformer的3D有效可扩展关注模型充分学习时间和空间维度之间的相关性，并引入方差估计来表征重建过程中的高频信息，实验结果表明该模型在视频SCI重建方面取得了最好的表现（比此前的SOTA算法PSNR提高了1.2dB）。

卷积-Transformer混合概率先验展开框架及其在视频快照压缩成像中的不确定性估计