Implicit neural representations (INR) excel in encoding videos within neural
networks, showcasing promise in computer vision tasks like video compression
and denoising. INR-based approaches reconstruct video frames from
content-agnostic embeddings, which hampers their efficacy in video frame
regression and restricts their generalization ability for video interpolation.
To address these deficiencies, Hybrid Neural Representation for Videos (HNeRV)
was introduced with content-adaptive embeddings. Nevertheless, HNeRV's
compression ratios remain relatively low, attributable to an oversight in
leveraging the network's shallow features and inter-frame residual information.
In this work, we introduce an advanced U-shaped architecture, Vector
Quantized-NeRV (VQ-NeRV), which integrates a novel component--the VQ-NeRV
Block. This block incorporates a codebook mechanism to discretize the network's
shallow residual features and inter-frame residual information effectively.
This approach proves particularly advantageous in video compression, as it
results in smaller size compared to quantized features. Furthermore, we
introduce an original codebook optimization technique, termed shallow codebook
optimization, designed to refine the utility and efficiency of the codebook.
The experimental evaluations indicate that VQ-NeRV outperforms HNeRV on video
regression tasks, delivering superior reconstruction quality (with an increase
of 1-2 dB in Peak Signal-to-Noise Ratio (PSNR)), better bit per pixel (bpp)
efficiency, and improved video inpainting outcomes.

通过引入具有自适应内容嵌入的混合神经表示视频（HNeRV）和一个新颖组件 ——VQ-NeRV 块，我们提出了一个先进的 U 型架构，Vector Quantized-NeRV（VQ-NeRV），以在视频回归任务中提供卓越的重建质量、更好的比特率和改善的视频修复结果。

VQ-NeRV：一种用于视频的向量量化神经表示

VQ-NeRV: A Vector Quantized Neural Representation for Videos

Implicit neural representations store videos as neural networks and have
performed well for various vision tasks such as video compression and
denoising. With frame index or positional index as input, implicit
representations (NeRV, E-NeRV, \etc) reconstruct video from fixed and
content-agnostic embeddings. Such embedding largely limits the regression
capacity and internal generalization for video interpolation. In this paper, we
propose a Hybrid Neural Representation for Videos (HNeRV), where a learnable
encoder generates content-adaptive embeddings, which act as the decoder input.
Besides the input embedding, we introduce HNeRV blocks, which ensure model
parameters are evenly distributed across the entire network, such that higher
layers (layers near the output) can have more capacity to store high-resolution
content and video details. With content-adaptive embeddings and re-designed
architecture, HNeRV outperforms implicit methods in video regression tasks for
both reconstruction quality ($+4.7$ PSNR) and convergence speed ($16\times$
faster), and shows better internal generalization. As a simple and efficient
video representation, HNeRV also shows decoding advantages for speed,
flexibility, and deployment, compared to traditional codecs~(H.264, H.265) and
learning-based compression methods. Finally, we explore the effectiveness of
HNeRV on downstream tasks such as video compression and video inpainting. We
provide project page at this https URL, and Code at
this https URL

本文提出了一种名为 HNeRV 的混合神经表示，其中可学习的编码器生成内容自适应嵌入，其用作解码器输入。通过内容自适应嵌入和重新设计的架构，HNeRV 在视频回归任务中的重构质量（+4.7 PSNR）和收敛速度（16 倍）方面优于隐式方法，并展示了更好的内部泛化。作为一种简单高效的视频表示方法，HNeRV 也与传统编解码器（H.264，H.265）和基于学习的压缩方法相比，具有更快的解码速度、更大的灵活性和更简单的部署，并且能够在视频压缩和视频修补等下游任务中发挥作用。