Extracting Implicit Neural Representations (INRs) on video data poses unique
challenges due to the additional temporal dimension. In the context of videos,
INRs have predominantly relied on a frame-only parameterization, which
sacrifices the spatiotemporal continuity observed in pixel-level (spatial)
representations. To mitigate this, we introduce Polynomial Neural
Representation for Videos (PNeRV), a parameter-wise efficient, patch-wise INR
for videos that preserves spatiotemporal continuity. PNeRV leverages the
modeling capabilities of Polynomial Neural Networks to perform the modulation
of a continuous spatial (patch) signal with a continuous time (frame) signal.
We further propose a custom Hierarchical Patch-wise Spatial Sampling Scheme
that ensures spatial continuity while retaining parameter efficiency. We also
employ a carefully designed Positional Embedding methodology to further enhance
PNeRV's performance. Our extensive experimentation demonstrates that PNeRV
outperforms the baselines in conventional Implicit Neural Representation tasks
like compression along with downstream applications that require spatiotemporal
continuity in the underlying representation. PNeRV not only addresses the
challenges posed by video data in the realm of INRs but also opens new avenues
for advanced video processing and analysis.

通过引入多项式神经网络，本研究提出了一种针对视频的参数高效、面片级内隐神经表示的方法（PNeRV），该方法在保留空间连续性的同时，充分利用了连续时间的模糊能力，解决了视频数据领域内内隐神经表示所面临的挑战，并为先进的视频处理和分析开辟了新的途径。

PNeRV: 用于视频的多项式神经表示

PNeRV: A Polynomial Neural Representation for Videos

Low-rank and sparse decomposition based methods find their use in many
applications involving background modeling such as clutter suppression and
object tracking. While Robust Principal Component Analysis (RPCA) has achieved
great success in performing this task, it can take hundreds of iterations to
converge and its performance decreases in the presence of different phenomena
such as occlusion, jitter and fast motion. The recently proposed deep unfolded
networks, on the other hand, have demonstrated better accuracy and improved
convergence over both their iterative equivalents as well as over other neural
network architectures. In this work, we propose a novel deep unfolded
spatiotemporal RPCA (DUST-RPCA) network, which explicitly takes advantage of
the spatial and temporal continuity in the low-rank component. Our experimental
results on the moving MNIST dataset indicate that DUST-RPCA gives better
accuracy when compared with the existing state of the art deep unfolded RPCA
networks.

本文提出了一种新的深度展开的时空鲁棒主成分分析神经网络方法 (DUST-RPCA)，该方法通过显式利用低秩成分中的时空连续性，在移动 MNIST 数据集上比现有的深度展开鲁棒主成分分析神经网络达到更好的精度。