Event cameras have emerged as a promising vision sensor in recent years due
to their unparalleled temporal resolution and dynamic range. While registration
of 2D RGB images to 3D point clouds is a long-standing problem in computer
vision, no prior work studies 2D-3D registration for event cameras. To this
end, we propose E2PNet, the first learning-based method for event-to-point
cloud registration. The core of E2PNet is a novel feature representation
network called Event-Points-to-Tensor (EP2T), which encodes event data into a
2D grid-shaped feature tensor. This grid-shaped feature enables matured
RGB-based frameworks to be easily used for event-to-point cloud registration,
without changing hyper-parameters and the training procedure. EP2T treats the
event input as spatio-temporal point clouds. Unlike standard 3D learning
architectures that treat all dimensions of point clouds equally, the novel
sampling and information aggregation modules in EP2T are designed to handle the
inhomogeneity of the spatial and temporal dimensions. Experiments on the MVSEC
and VECtor datasets demonstrate the superiority of E2PNet over hand-crafted and
other learning-based methods. Compared to RGB-based registration, E2PNet is
more robust to extreme illumination or fast motion due to the use of event
data. Beyond 2D-3D registration, we also show the potential of EP2T for other
vision tasks such as flow estimation, event-to-image reconstruction and object
recognition. The source code can be found at:
this https URL

2D-3D registration for event cameras is studied using a learning-based method called E2PNet, which employs a novel feature representation network called EP2T. E2PNet outperforms hand-crafted and other learning-based methods, and demonstrates potential for various vision tasks.