The amplification of high-speed micro-motions holds significant promise, with
applications spanning fault detection in fast-paced industrial environments to
refining precision in medical procedures. However, conventional motion
magnification algorithms often encounter challenges in high-speed scenarios due
to low sampling rates or motion blur. In recent years, spike cameras have
emerged as a superior alternative for visual tasks in such environments, owing
to their unique capability to capture temporal and spatial frequency domains
with exceptional fidelity. Unlike conventional cameras, which operate at fixed,
low frequencies, spike cameras emulate the functionality of the retina,
asynchronously capturing photon changes at each pixel position using spike
streams. This innovative approach comprehensively records temporal and spatial
visual information, rendering it particularly suitable for magnifying
high-speed micro-motions.This paper introduces SpikeMM, a pioneering
spike-based algorithm tailored specifically for high-speed motion
magnification. SpikeMM integrates multi-level information extraction, spatial
upsampling, and motion magnification modules, offering a self-supervised
approach adaptable to a wide range of scenarios. Notably, SpikeMM facilitates
seamless integration with high-performance super-resolution and motion
magnification algorithms. We substantiate the efficacy of SpikeMM through
rigorous validation using scenes captured by spike cameras, showcasing its
capacity to magnify motions in real-world high-frequency settings.

SpikeMM 是一种基于 spike 的算法，专门为高速运动放大而设计，通过多级信息提取、空间上采样和运动放大模块的集成，实现了自我监督方法适用于各种场景，并与高性能超分辨率和运动放大算法无缝结合，经验证在实际高频设置中放大运动的能力。

SpikeMM：高速微运动的弹性放大

SpikeMM: Flexi-Magnification of High-Speed Micro-Motions

Spike cameras, with their exceptional temporal resolution, are
revolutionizing high-speed visual applications. Large-scale synthetic datasets
have significantly accelerated the development of these cameras, particularly
in reconstruction and optical flow. However, current synthetic datasets for
spike cameras lack sophistication. Addressing this gap, we introduce SCSim, a
novel and more realistic spike camera simulator with a comprehensive noise
model. SCSim is adept at autonomously generating driving scenarios and
synthesizing corresponding spike streams. To enhance the fidelity of these
streams, we've developed a comprehensive noise model tailored to the unique
circuitry of spike cameras. Our evaluations demonstrate that SCSim outperforms
existing simulation methods in generating authentic spike streams. Crucially,
SCSim simplifies the creation of datasets, thereby greatly advancing
spike-based visual tasks like reconstruction. Our project refers to
this https URL

通过 SCSim 模拟器生成更真实的尖峰相机图像，其中包括了新的噪声模型，从而提高了重建任务等尖峰相机视觉任务的性能。

SCSim：一个现实的脉冲相机模拟器

SCSim: A Realistic Spike Cameras Simulator

Spike cameras, leveraging spike-based integration sampling and high temporal
resolution, offer distinct advantages over standard cameras. However, existing
approaches reliant on spike cameras often assume optimal illumination, a
condition frequently unmet in real-world scenarios. To address this, we
introduce SpikeNeRF, the first work that derives a NeRF-based volumetric scene
representation from spike camera data. Our approach leverages NeRF's multi-view
consistency to establish robust self-supervision, effectively eliminating
erroneous measurements and uncovering coherent structures within exceedingly
noisy input amidst diverse real-world illumination scenarios. The framework
comprises two core elements: a spike generation model incorporating an
integrate-and-fire neuron layer and parameters accounting for non-idealities,
such as threshold variation, and a spike rendering loss capable of generalizing
across varying illumination conditions. We describe how to effectively optimize
neural radiance fields to render photorealistic novel views from the novel
continuous spike stream, demonstrating advantages over other vision sensors in
certain scenes. Empirical evaluations conducted on both real and novel
realistically simulated sequences affirm the efficacy of our methodology. The
dataset and source code are released at
this https URL

使用尖峰相机数据，利用 SpikeNeRF 从尖峰相机数据中得出了基于 NeRF 的体积场景表示，既能在真实世界中去除错误测量，又能揭示出在各种真实世界的照明情况下一贯的结构，从而在某些场景中展现出比其他视觉传感器更大的优势。

SpikeNeRF：从连续脉冲流中学习神经辐射场

SpikeNeRF: Learning Neural Radiance Fields from Continuous Spike Stream

Reconstructing a sequence of sharp images from the blurry input is crucial
for enhancing our insights into the captured scene and poses a significant
challenge due to the limited temporal features embedded in the image. Spike
cameras, sampling at rates up to 40,000 Hz, have proven effective in capturing
motion features and beneficial for solving this ill-posed problem. Nonetheless,
existing methods fall into the supervised learning paradigm, which suffers from
notable performance degradation when applied to real-world scenarios that
diverge from the synthetic training data domain. Moreover, the quality of
reconstructed images is capped by the generated images based on motion analysis
interpolation, which inherently differs from the actual scene, affecting the
generalization ability of these methods in real high-speed scenarios. To
address these challenges, we propose the first self-supervised framework for
the task of spike-guided motion deblurring. Our approach begins with the
formulation of a spike-guided deblurring model that explores the theoretical
relationships among spike streams, blurry images, and their corresponding sharp
sequences. We subsequently develop a self-supervised cascaded framework to
alleviate the issues of spike noise and spatial-resolution mismatching
encountered in the deblurring model. With knowledge distillation and
re-blurring loss, we further design a lightweight deblur network to generate
high-quality sequences with brightness and texture consistency with the
original input. Quantitative and qualitative experiments conducted on our
real-world and synthetic datasets with spikes validate the superior
generalization of the proposed framework. Our code, data and trained models
will be available at https://github.com/chenkang455/S-SDM.

提出了第一个自监督框架来解决基于脉冲引导的运动去模糊任务，通过理论关系和知识蒸馏设计轻量级去模糊网络，生成与原始输入亮度和纹理一致的高质量序列。