Depth estimation is crucial for interpreting complex environments, especially
in areas such as autonomous vehicle navigation and robotics. Nonetheless,
obtaining accurate depth readings from event camera data remains a formidable
challenge. Event cameras operate differently from traditional digital cameras,
continuously capturing data and generating asynchronous binary spikes that
encode time, location, and light intensity. Yet, the unique sampling mechanisms
of event cameras render standard image based algorithms inadequate for
processing spike data. This necessitates the development of innovative,
spike-aware algorithms tailored for event cameras, a task compounded by the
irregularity, continuity, noise, and spatial and temporal characteristics
inherent in spiking data.Harnessing the strong generalization capabilities of
transformer neural networks for spatiotemporal data, we propose a purely
spike-driven spike transformer network for depth estimation from spiking camera
data. To address performance limitations with Spiking Neural Networks (SNN), we
introduce a novel single-stage cross-modality knowledge transfer framework
leveraging knowledge from a large vision foundational model of artificial
neural networks (ANN) (DINOv2) to enhance the performance of SNNs with limited
data. Our experimental results on both synthetic and real datasets show
substantial improvements over existing models, with notable gains in Absolute
Relative and Square Relative errors (49% and 39.77% improvements over the
benchmark model Spike-T, respectively). Besides accuracy, the proposed model
also demonstrates reduced power consumptions, a critical factor for practical
applications.

基于脉冲相机数据的纯脉冲驱动脉冲变压器网络模型以及跨模态知识迁移框架，在深度估计领域取得了显著提升的结果，同时减少了功耗消耗。

基于跨模态知识蒸馏的事件相机深度估计的新型脉冲变压器网络

A Novel Spike Transformer Network for Depth Estimation from Event  Cameras via Cross-modality Knowledge Distillation

We introduce a method to train Quantized Neural Networks (QNNs) --- neural
networks with extremely low precision (e.g., 1-bit) weights and activations, at
run-time. At train-time the quantized weights and activations are used for
computing the parameter gradients. During the forward pass, QNNs drastically
reduce memory size and accesses, and replace most arithmetic operations with
bit-wise operations. As a result, power consumption is expected to be
drastically reduced. We trained QNNs over the MNIST, CIFAR-10, SVHN and
ImageNet datasets. The resulting QNNs achieve prediction accuracy comparable to
their 32-bit counterparts. For example, our quantized version of AlexNet with
1-bit weights and 2-bit activations achieves $51\%$ top-1 accuracy. Moreover,
we quantize the parameter gradients to 6-bits as well which enables gradients
computation using only bit-wise operation. Quantized recurrent neural networks
were tested over the Penn Treebank dataset, and achieved comparable accuracy as
their 32-bit counterparts using only 4-bits. Last but not least, we programmed
a binary matrix multiplication GPU kernel with which it is possible to run our
MNIST QNN 7 times faster than with an unoptimized GPU kernel, without suffering
any loss in classification accuracy. The QNN code is available online.

本研究介绍了一种训练低精度神经网络的方法，该方法使用二进制操作代替计算，达到降低内存大小、减少电力消耗的目的。经过 MNIST、CIFAR-10、SVHN、ImageNet 和 Penn Treebank 数据集的测试，结果表明 1 位权重和 2 位激活的量化版本的 AlexNet 能够达到 51% 的准确率，训练过程中也能使用仅有的二进制操作实现损失函数的计算，并在损失部分的代码上进行了优化，使得 QNN 的运行速度能比未优化的 GPU 加速速度快七倍，并且没有影响分类准确性。