The stereo event-intensity camera setup is widely applied to leverage the advantages of both event cameras with low latency and intensity cameras that capture accurate brightness and texture information. However, such a setup commonly encounters cross-modality parallax that is difficult to be eliminated solely with stereo rectification especially for real-world scenes with complex motions and varying depths, posing artifacts and distortion for existing Event-based Video Frame Interpolation (E-VFI) approaches. To tackle this problem, we propose a novel Stereo Event-based VFI (SE-VFI) network (SEVFI-Net) to generate high-quality intermediate frames and corresponding disparities from misaligned inputs consisting of two consecutive keyframes and event streams emitted between them. Specifically, we propose a Feature Aggregation Module (FAM) to alleviate the parallax and achieve spatial alignment in the feature domain. We then exploit the fused features accomplishing accurate optical flow and disparity estimation, and achieving better interpolated results through flow-based and synthesis-based ways. We also build a stereo visual acquisition system composed of an event camera and an RGB-D camera to collect a new Stereo Event-Intensity Dataset (SEID) containing diverse scenes with complex motions and varying depths. Experiments on public real-world stereo datasets, i.e., DSEC and MVSEC, and our SEID dataset demonstrate that our proposed SEVFI-Net outperforms state-of-the-art methods by a large margin.

本文提出了一种新的立体事件驱动视频帧插值（SE-VFI）网络（SEVFI-Net），通过特征聚合模块（FAM）解决交叉模态视差，利用融合特征实现精确的光流和视差估计，并通过基于流和基于合成的方式实现更好的插值结果。同时，作者构建了一个由事件相机和RGB-D相机组成的立体视觉采集系统，收集了一个新的立体事件强度数据集（SEID），实验证明SEVFI-Net在公共真实世界立体数据集（DSEC和MVSEC）和作者的SEID数据集上优于现有方法。

使用立体事件和强度相机进行视频帧插值