In recent years, end-to-end learnt video codecs have demonstrated their
potential to compete with conventional coding algorithms in term of compression
efficiency. However, most learning-based video compression models are
associated with high computational complexity and latency, in particular at the
decoder side, which limits their deployment in practical applications. In this
paper, we present a novel model-agnostic pruning scheme based on gradient decay
and adaptive layer-wise distillation. Gradient decay enhances parameter
exploration during sparsification whilst preventing runaway sparsity and is
superior to the standard Straight-Through Estimation. The adaptive layer-wise
distillation regulates the sparse training in various stages based on the
distortion of intermediate features. This stage-wise design efficiently updates
parameters with minimal computational overhead. The proposed approach has been
applied to three popular end-to-end learnt video codecs, FVC, DCVC, and
DCVC-HEM. Results confirm that our method yields up to 65% reduction in MACs
and 2x speed-up with less than 0.3dB drop in BD-PSNR. Supporting code and
supplementary material can be downloaded from:
this https URL

最近几年，端到端学习的视频编码器在压缩效率方面已经显示出与传统编码算法竞争的潜力。然而，大多数基于学习的视频压缩模型与高计算复杂性和延迟相关，特别是在解码器方面，这限制了它们在实际应用中的部署。本文提出了一种基于梯度衰减和自适应分层蒸馏的新型模型不可知剪枝方案。梯度衰减增强了稀疏化过程中的参数探索，同时防止过度稀疏，比标准的直通估计方法更优。自适应分层蒸馏根据中间特征的失真情况，在不同阶段有效地调整稀疏训练，以最小的计算开销更新参数。所提出的方法已应用于三种流行的端到端学习的视频编码器，即 FVC、DCVC 和 DCVC-HEM。结果表明，我们的方法在 MACS 上减少了高达 65％，速度提升了 2 倍，并且 BD-PSNR 下降不到 0.3dB。支持代码和补充材料可从此 https URL 下载。

梯度衰减和逐层蒸馏加速学习视频编解码器

Accelerating Learnt Video Codecs with Gradient Decay and Layer-wise  Distillation

In the past few years, Softmax has become a common component in neural
network frameworks. In this paper, a gradient decay hyperparameter is
introduced in Softmax to control the probability-dependent gradient decay rate
during training. By following the theoretical analysis and empirical results of
a variety of model architectures trained on MNIST, CIFAR-10/100 and SVHN, we
find that the generalization performance depends significantly on the gradient
decay rate as the confidence probability rises, i.e., the gradient decreases
convexly or concavely as the sample probability increases. Moreover,
optimization with the small gradient decay shows a similar curriculum learning
sequence where hard samples are in the spotlight only after easy samples are
convinced sufficiently, and well-separated samples gain a higher gradient to
reduce intra-class distance. Based on the analysis results, we can provide
evidence that the large margin Softmax will affect the local Lipschitz
constraint of the loss function by regulating the probability-dependent
gradient decay rate. This paper provides a new perspective and understanding of
the relationship among concepts of large margin Softmax, local Lipschitz
constraint and curriculum learning by analyzing the gradient decay rate.
Besides, we propose a warm-up strategy to dynamically adjust Softmax loss in
training, where the gradient decay rate increases from over-small to speed up
the convergence rate.

本文探究了在神经网络框架中常用的 Softmax，在训练过程中引入渐变衰减超参数控制概率相关的梯度衰减率。作者通过实验发现，梯度衰减率随置信概率上升而凸凹变化，且使用小梯度衰减的优化方式表现出类似于课程学习的顺序，可以说明较大间距的 Softmax 将影响损失函数的局部 Lipschitz 约束。在分析结果的基础上，作者提出了一种 “热身策略” 来动态调整训练中的 Softmax 损失，从而加快收敛速度。