Though many compilation and runtime systems have been developed for DNNs in
recent years, the focus has largely been on static DNNs. Dynamic DNNs, where
tensor shapes and sizes and even the set of operators used are dependent upon
the input and/or execution, are becoming common. This paper presents SoD$^2$, a
comprehensive framework for optimizing Dynamic DNNs. The basis of our approach
is a classification of common operators that form DNNs, and the use of this
classification towards a Rank and Dimension Propagation (RDP) method. This
framework statically determines the shapes of operators as known constants,
symbolic constants, or operations on these. Next, using RDP we enable a series
of optimizations, like fused code generation, execution (order) planning, and
even runtime memory allocation plan generation. By evaluating the framework on
10 emerging Dynamic DNNs and comparing it against several existing systems, we
demonstrate both reductions in execution latency and memory requirements, with
RDP-enabled key optimizations responsible for much of the gains. Our evaluation
results show that SoD$^2$ runs up to $3.9\times$ faster than these systems
while saving up to $88\%$ peak memory consumption.

SoD$^2$ 是一个用于优化动态深度神经网络的综合框架，通过分类和使用常见操作符，结合 Rank and Dimension Propagation (RDP) 方法，实现了静态确定操作符形状并进行一系列优化，从而显著减少执行延迟和内存需求。

SoD$^2$: 静态优化动态深度神经网络

SoD$^2$: Statically Optimizing Dynamic Deep Neural Network

Mobile and embedded platforms are increasingly required to efficiently
execute computationally demanding DNNs across heterogeneous processing
elements. At runtime, the available hardware resources to DNNs can vary
considerably due to other concurrently running applications. The performance
requirements of the applications could also change under different scenarios.
To achieve the desired performance, dynamic DNNs have been proposed in which
the number of channels/layers can be scaled in real time to meet different
requirements under varying resource constraints. However, the training process
of such dynamic DNNs can be costly, since platform-aware models of different
deployment scenarios must be retrained to become dynamic. This paper proposes
Dynamic-OFA, a novel dynamic DNN approach for state-of-the-art platform-aware
NAS models (i.e. Once-for-all network (OFA)). Dynamic-OFA pre-samples a family
of sub-networks from a static OFA backbone model, and contains a runtime
manager to choose different sub-networks under different runtime environments.
As such, Dynamic-OFA does not need the traditional dynamic DNN training
pipeline. Compared to the state-of-the-art, our experimental results using
ImageNet on a Jetson Xavier NX show that the approach is up to 3.5x (CPU), 2.4x
(GPU) faster for similar ImageNet Top-1 accuracy, or 3.8% (CPU), 5.1% (GPU)
higher accuracy at similar latency.

本研究论文探讨了动态深度神经网络在不同运行环境下实现快速性能的问题，并提出了一种新的动态 DNN 方法 Dynamic-OFA，该方法通过预先采样 OFA 模型中的子网，配合运行时管理器在不同的运行环境中选择不同的子网来实现。在实验中，使用 ImageNet 数据集验证了该方法的性能表现，相比已有方法，Dynamic-OFA 在 Jetson Xavier NX 上可以实现最高 3.5x 的 CPU 速度和 2.4x 的 GPU 速度提升，并相应地实现 3.8％的 CPU 精度和 5.1％的 GPU 精度提升。