Deep neural networks (DNNs) are frequently employed in a variety of computer
vision applications. Nowadays, an emerging trend in the current video
distribution system is to take advantage of DNN's overfitting properties to
perform video resolution upscaling. By splitting videos into chunks and
applying a super-resolution (SR) model to overfit each chunk, this scheme of SR
models plus video chunks is able to replace traditional video transmission to
enhance video quality and transmission efficiency. However, many models and
chunks are needed to guarantee high performance, which leads to tremendous
overhead on model switching and memory footprints at the user end. To resolve
such problems, we propose a Dynamic Deep neural network assisted by a
Content-Aware data processing pipeline to reduce the model number down to one
(Dy-DCA), which helps promote performance while conserving computational
resources. Additionally, to achieve real acceleration on the user end, we
designed a framework that optimizes dynamic features (e.g., dynamic shapes,
sizes, and control flow) in Dy-DCA to enable a series of compilation
optimizations, including fused code generation, static execution planning, etc.
By employing such techniques, our method achieves better PSNR and real-time
performance (33 FPS) on an off-the-shelf mobile phone. Meanwhile, assisted by
our compilation optimization, we achieve a 1.7$\times$ speedup while saving up
to 1.61$\times$ memory consumption. Code available in
this https URL

通过动态深度神经网络辅助的内容感知数据处理管道，我们提出了一种减少模型数量的方法，以提高性能并节省计算资源，并通过编译优化实现更好的 PSNR 和实时性能，在现成的手机上实现了 1.7 倍的加速，节省了高达 1.61 倍的内存消耗。

基于动态算法和编译器协同设计的设备端超分辨率中的数据过拟合

Data Overfitting for On-Device Super-Resolution with Dynamic Algorithm  and Compiler Co-Design

As deep convolutional neural networks (DNNs) are widely used in various
fields of computer vision, leveraging the overfitting ability of the DNN to
achieve video resolution upscaling has become a new trend in the modern video
delivery system. By dividing videos into chunks and overfitting each chunk with
a super-resolution model, the server encodes videos before transmitting them to
the clients, thus achieving better video quality and transmission efficiency.
However, a large number of chunks are expected to ensure good overfitting
quality, which substantially increases the storage and consumes more bandwidth
resources for data transmission. On the other hand, decreasing the number of
chunks through training optimization techniques usually requires high model
capacity, which significantly slows down execution speed. To reconcile such, we
propose a novel method for high-quality and efficient video resolution
upscaling tasks, which leverages the spatial-temporal information to accurately
divide video into chunks, thus keeping the number of chunks as well as the
model size to minimum. Additionally, we advance our method into a single
overfitting model by a data-aware joint training technique, which further
reduces the storage requirement with negligible quality drop. We deploy our
models on an off-the-shelf mobile phone, and experimental results show that our
method achieves real-time video super-resolution with high video quality.
Compared with the state-of-the-art, our method achieves 28 fps streaming speed
with 41.6 PSNR, which is 14$\times$ faster and 2.29 dB better in the live video
resolution upscaling tasks. Our codes are available at:
this https URL

利用深度卷积神经网络，结合时空信息，通过数据导向联合训练技术，实现高质量高效实时的视频超分辨率提升，相比现有技术在流媒体视频分辨率提升任务中提高了 14 倍帧率同时提高 2.29dB 的画质。