Event cameras and RGB cameras exhibit complementary characteristics in
imaging: the former possesses high dynamic range (HDR) and high temporal
resolution, while the latter provides rich texture and color information. This
makes the integration of event cameras into middle- and high-level RGB-based
vision tasks highly promising. However, challenges arise in multi-modal fusion,
data annotation, and model architecture design. In this paper, we propose
EvPlug, which learns a plug-and-play event and image fusion module from the
supervision of the existing RGB-based model. The learned fusion module
integrates event streams with image features in the form of a plug-in, endowing
the RGB-based model to be robust to HDR and fast motion scenes while enabling
high temporal resolution inference. Our method only requires unlabeled
event-image pairs (no pixel-wise alignment required) and does not alter the
structure or weights of the RGB-based model. We demonstrate the superiority of
EvPlug in several vision tasks such as object detection, semantic segmentation,
and 3D hand pose estimation

该论文提出了 EvPlug，一种能够将事件相机与图像相结合的方法，在不需要像素级对齐的情况下，通过学习融合模块的方式，使得 RGB 模型在高动态范围和快速运动场景下具备鲁棒性和高时序分辨率推理能力，从而在物体检测、语义分割和 3D 手姿估计等多个视觉任务中展现了优越性。