The bio-inspired event cameras or dynamic vision sensors are capable of
asynchronously capturing per-pixel brightness changes (called event-streams) in
high temporal resolution and high dynamic range. However, the non-structural
spatial-temporal event-streams make it challenging for providing intuitive
visualization with rich semantic information for human vision. It calls for
events-to-video (E2V) solutions which take event-streams as input and generate
high quality video frames for intuitive visualization. However, current
solutions are predominantly data-driven without considering the prior knowledge
of the underlying statistics relating event-streams and video frames. It highly
relies on the non-linearity and generalization capability of the deep neural
networks, thus, is struggling on reconstructing detailed textures when the
scenes are complex. In this work, we propose \textbf{E2HQV}, a novel E2V
paradigm designed to produce high-quality video frames from events. This
approach leverages a model-aided deep learning framework, underpinned by a
theory-inspired E2V model, which is meticulously derived from the fundamental
imaging principles of event cameras. To deal with the issue of state-reset in
the recurrent components of E2HQV, we also design a temporal shift embedding
module to further improve the quality of the video frames. Comprehensive
evaluations on the real world event camera datasets validate our approach, with
E2HQV, notably outperforming state-of-the-art approaches, e.g., surpassing the
second best by over 40\% for some evaluation metrics.

提出了一种名为 E2HQV 的新型 E2V 范例，该范例利用了模型辅助的深度学习框架，从基于事件相机的基本成像原理中精心推导出的理论驱动的 E2V 模型，以及为了进一步提高视频帧质量而设计的时间偏移嵌入模块。对真实世界事件相机数据集的综合评估验证了我们的方法的有效性，E2HQV 在一些评估指标上超过了其他最先进的方法超过 40%。

E2HQV：通过理论启发的模型辅助深度学习从事件相机生成高质量视频

E2HQV: High-Quality Video Generation from Event Camera via  Theory-Inspired Model-Aided Deep Learning

Traffic object detection under variable illumination is challenging due to
the information loss caused by the limited dynamic range of conventional
frame-based cameras. To address this issue, we introduce bio-inspired event
cameras and propose a novel Structure-aware Fusion Network (SFNet) that
extracts sharp and complete object structures from the event stream to
compensate for the lost information in images through cross-modality fusion,
enabling the network to obtain illumination-robust representations for traffic
object detection. Specifically, to mitigate the sparsity or blurriness issues
arising from diverse motion states of traffic objects in fixed-interval event
sampling methods, we propose the Reliable Structure Generation Network (RSGNet)
to generate Speed Invariant Frames (SIF), ensuring the integrity and sharpness
of object structures. Next, we design a novel Adaptive Feature Complement
Module (AFCM) which guides the adaptive fusion of two modality features to
compensate for the information loss in the images by perceiving the global
lightness distribution of the images, thereby generating illumination-robust
representations. Finally, considering the lack of large-scale and high-quality
annotations in the existing event-based object detection datasets, we build a
DSEC-Det dataset, which consists of 53 sequences with 63,931 images and more
than 208,000 labels for 8 classes. Extensive experimental results demonstrate
that our proposed SFNet can overcome the perceptual boundaries of conventional
cameras and outperform the frame-based method by 8.0% in mAP50 and 5.9% in
mAP50:95. Our code and dataset will be available at
this https URL

针对传统基于帧的摄像机由于动态范围受限而导致的信息丢失，我们引入了生物启发式事件摄像机，并提出了一种新颖的结构感知融合网络（SFNet），通过跨模态融合从事件流中提取清晰和完整的物体结构，以补偿图像中的信息丢失，从而使网络能够获得对光照鲁棒的交通物体检测表示。

基于 RGB-Event 融合的可变光照下的交通物体检测增强

Enhancing Traffic Object Detection in Variable Illumination with  RGB-Event Fusion

Tracking using bio-inspired event cameras has drawn more and more attention
in recent years. Existing works either utilize aligned RGB and event data for
accurate tracking or directly learn an event-based tracker. The first category
needs more cost for inference and the second one may be easily influenced by
noisy events or sparse spatial resolution. In this paper, we propose a novel
hierarchical knowledge distillation framework that can fully utilize
multi-modal / multi-view information during training to facilitate knowledge
transfer, enabling us to achieve high-speed and low-latency visual tracking
during testing by using only event signals. Specifically, a teacher
Transformer-based multi-modal tracking framework is first trained by feeding
the RGB frame and event stream simultaneously. Then, we design a new
hierarchical knowledge distillation strategy which includes pairwise
similarity, feature representation, and response maps-based knowledge
distillation to guide the learning of the student Transformer network.
Moreover, since existing event-based tracking datasets are all low-resolution
($346 \times 260$), we propose the first large-scale high-resolution ($1280
\times 720$) dataset named EventVOT. It contains 1141 videos and covers a wide
range of categories such as pedestrians, vehicles, UAVs, ping pongs, etc.
Extensive experiments on both low-resolution (FE240hz, VisEvent, COESOT), and
our newly proposed high-resolution EventVOT dataset fully validated the
effectiveness of our proposed method. The dataset, evaluation toolkit, and
source code are available on
https://github.com/Event-AHU/EventVOT_Benchmark

通过多模态跟踪和知识蒸馏方法，在高分辨率事件信号下实现了高速低延迟的视觉跟踪的研究。