The extraction of a clean background image by removing foreground occlusion
holds immense practical significance, but it also presents several challenges.
Presently, the majority of de-occlusion research focuses on addressing this
issue through the extraction and synthesis of discrete images from calibrated
camera arrays. Nonetheless, the restoration quality tends to suffer when faced
with dense occlusions or high-speed motions due to limited perspectives and
motion blur. To successfully remove dense foreground occlusion, an effective
multi-view visual information integration approach is required. Introducing the
spike camera as a novel type of neuromorphic sensor offers promising
capabilities with its ultra-high temporal resolution and high dynamic range. In
this paper, we propose an innovative solution for tackling the de-occlusion
problem through continuous multi-view imaging using only one spike camera
without any prior knowledge of camera intrinsic parameters and camera poses. By
rapidly moving the spike camera, we continually capture the dense stream of
spikes from the occluded scene. To process the spikes, we build a novel model
\textbf{SpkOccNet}, in which we integrate information of spikes from continuous
viewpoints within multi-windows, and propose a novel cross-view mutual
attention mechanism for effective fusion and refinement. In addition, we
contribute the first real-world spike-based dataset \textbf{S-OCC} for
occlusion removal. The experimental results demonstrate that our proposed model
efficiently removes dense occlusions in diverse scenes while exhibiting strong
generalization.

本文提出了一种使用仅一个 SPIKE 相机进行连续多视角成像的创新解决方案，即 SpkOccNet，利用跨视图注意机制有效融合并优化多窗口内连续视图的 SPIKE 信息，实现在不知道相机内参数和相机姿态的情况下高效去除多种场景下的密集遮挡，取得了良好的实验效果。

揭示尖峰流在从密集连续视图中消除前景遮挡方面的潜力

Unveiling the Potential of Spike Streams for Foreground Occlusion  Removal from Densely Continuous Views

In recent decades, 3D morphable model (3DMM) has been commonly used in
image-based photorealistic 3D face reconstruction. However, face images are
often corrupted by serious occlusion by non-face objects including eyeglasses,
masks, and hands. Such objects block the correct capture of landmarks and
shading information. Therefore, the reconstructed 3D face model is hardly
reusable. In this paper, a novel method is proposed to restore de-occluded face
images based on inverse use of 3DMM and generative adversarial network. We
utilize the 3DMM prior to the proposed adversarial network and combine a global
and local adversarial convolutional neural network to learn face de-occlusion
model. The 3DMM serves not only as geometric prior but also proposes the face
region for the local discriminator. Experiment results confirm the
effectiveness and robustness of the proposed algorithm in removing challenging
types of occlusions with various head poses and illumination. Furthermore, the
proposed method reconstructs the correct 3D face model with de-occluded
textures.

本研究提出了一种新颖的基于 3DMM 和生成对抗网络的方法来恢复去遮挡的人脸图像并重建正确的 3D 面部模型，实验结果证实了该算法在消除各种头部姿势和光照等难以处理的遮挡方面的有效性和鲁棒性。