Depth estimation is crucial for interpreting complex environments, especially
in areas such as autonomous vehicle navigation and robotics. Nonetheless,
obtaining accurate depth readings from event camera data remains a formidable
challenge. Event cameras operate differently from traditional digital cameras,
continuously capturing data and generating asynchronous binary spikes that
encode time, location, and light intensity. Yet, the unique sampling mechanisms
of event cameras render standard image based algorithms inadequate for
processing spike data. This necessitates the development of innovative,
spike-aware algorithms tailored for event cameras, a task compounded by the
irregularity, continuity, noise, and spatial and temporal characteristics
inherent in spiking data.Harnessing the strong generalization capabilities of
transformer neural networks for spatiotemporal data, we propose a purely
spike-driven spike transformer network for depth estimation from spiking camera
data. To address performance limitations with Spiking Neural Networks (SNN), we
introduce a novel single-stage cross-modality knowledge transfer framework
leveraging knowledge from a large vision foundational model of artificial
neural networks (ANN) (DINOv2) to enhance the performance of SNNs with limited
data. Our experimental results on both synthetic and real datasets show
substantial improvements over existing models, with notable gains in Absolute
Relative and Square Relative errors (49% and 39.77% improvements over the
benchmark model Spike-T, respectively). Besides accuracy, the proposed model
also demonstrates reduced power consumptions, a critical factor for practical
applications.

基于脉冲相机数据的纯脉冲驱动脉冲变压器网络模型以及跨模态知识迁移框架，在深度估计领域取得了显著提升的结果，同时减少了功耗消耗。

基于跨模态知识蒸馏的事件相机深度估计的新型脉冲变压器网络

A Novel Spike Transformer Network for Depth Estimation from Event  Cameras via Cross-modality Knowledge Distillation

This paper introduces a self-supervised learning framework designed for
pre-training neural networks tailored to dense prediction tasks using event
camera data. Our approach utilizes solely event data for training.
Transferring achievements from dense RGB pre-training directly to event
camera data yields subpar performance. This is attributed to the spatial
sparsity inherent in an event image (converted from event data), where many
pixels do not contain information. To mitigate this sparsity issue, we encode
an event image into event patch features, automatically mine contextual
similarity relationships among patches, group the patch features into
distinctive contexts, and enforce context-to-context similarities to learn
discriminative event features.
For training our framework, we curate a synthetic event camera dataset
featuring diverse scene and motion patterns. Transfer learning performance on
downstream dense prediction tasks illustrates the superiority of our method
over state-of-the-art approaches. Notably, our single model secured the top
position in the challenging DSEC-Flow benchmark.

本文介绍了一种自监督学习框架，用于预训练针对密集预测任务的神经网络，该框架使用事件相机数据进行训练，并在转移学习方面展现出卓越性能。

事件相机数据密集预训练

Event Camera Data Dense Pre-training

Recent advances in 2D zero-shot and few-shot recognition often leverage large
pre-trained vision-language models (VLMs) such as CLIP. Due to a shortage of
suitable datasets, it is currently infeasible to train such models for event
camera data. Thus, leveraging existing models across modalities is an important
research challenge. In this work, we propose EventCLIP, a new method that
utilizes CLIP for zero-shot and few-shot recognition on event camera data.
First, we demonstrate the suitability of CLIP's image embeddings for zero-shot
event classification by converting raw events to 2D grid-based representations.
Second, we propose a feature adapter that aggregates temporal information over
event frames and refines text embeddings to better align with the visual
inputs. We evaluate our work on N-Caltech, N-Cars, and N-ImageNet datasets
under the few-shot learning setting, where EventCLIP achieves state-of-the-art
performance. Finally, we show that the robustness of existing event-based
classifiers against data variations can be further boosted by ensembling with
EventCLIP.

本研究提出了一种新的方法：EventCLIP，利用 2D zero-shot recognition 和 few-shot learning 来识别 event camera data，并且使用 temporal feature adapter 来提高模型性能。通过实验证明，EventCLIP 在不同数据集上表现出 state-of-the-art performance，并且能够提高现有的 event-based classifiers 对数据变化的鲁棒性。