Recently quanta image sensors (QIS) -- ultra-fast, zero-read-noise binary
image sensors -- have demonstrated remarkable imaging capabilities in many
challenging scenarios. Despite their potential, the adoption of these sensors
is severely hampered by (a) high data rates and (b) the need for new
computational pipelines to handle the unconventional raw data. We introduce a
simple, low-bandwidth computational pipeline to address these challenges. Our
approach is based on a novel streaming representation with a small memory
footprint, efficiently capturing intensity information at multiple temporal
scales. Updating the representation requires only 16 floating-point
operations/pixel, which can be efficiently computed online at the native frame
rate of the binary frames. We use a neural network operating on this
representation to reconstruct videos in real-time (10-30 fps). We illustrate
why such representation is well-suited for these emerging sensors, and how it
offers low latency and high frame rate while retaining flexibility for
downstream computer vision. Our approach results in significant data bandwidth
reductions ~100X and real-time image reconstruction and computer vision --
$10^4$-$10^5$ reduction in computation than existing state-of-the-art approach
while maintaining comparable quality. To the best of our knowledge, our
approach is the first to achieve online, real-time image reconstruction on QIS.

我们引入了一个简单、低带宽的计算流程来解决快速量子图像传感器的数据率和处理非传统原始数据的挑战，并使用新型的流式表示和神经网络来在线实时重建图像。