Implicit neural representations store videos as neural networks and have
performed well for various vision tasks such as video compression and
denoising. With frame index or positional index as input, implicit
representations (NeRV, E-NeRV, \etc) reconstruct video from fixed and
content-agnostic embeddings. Such embedding largely limits the regression
capacity and internal generalization for video interpolation. In this paper, we
propose a Hybrid Neural Representation for Videos (HNeRV), where a learnable
encoder generates content-adaptive embeddings, which act as the decoder input.
Besides the input embedding, we introduce HNeRV blocks, which ensure model
parameters are evenly distributed across the entire network, such that higher
layers (layers near the output) can have more capacity to store high-resolution
content and video details. With content-adaptive embeddings and re-designed
architecture, HNeRV outperforms implicit methods in video regression tasks for
both reconstruction quality ($+4.7$ PSNR) and convergence speed ($16\times$
faster), and shows better internal generalization. As a simple and efficient
video representation, HNeRV also shows decoding advantages for speed,
flexibility, and deployment, compared to traditional codecs~(H.264, H.265) and
learning-based compression methods. Finally, we explore the effectiveness of
HNeRV on downstream tasks such as video compression and video inpainting. We
provide project page at this https URL, and Code at
this https URL

本文提出了一种名为 HNeRV 的混合神经表示，其中可学习的编码器生成内容自适应嵌入，其用作解码器输入。通过内容自适应嵌入和重新设计的架构，HNeRV 在视频回归任务中的重构质量（+4.7 PSNR）和收敛速度（16 倍）方面优于隐式方法，并展示了更好的内部泛化。作为一种简单高效的视频表示方法，HNeRV 也与传统编解码器（H.264，H.265）和基于学习的压缩方法相比，具有更快的解码速度、更大的灵活性和更简单的部署，并且能够在视频压缩和视频修补等下游任务中发挥作用。