There has been many papers in academic literature on quantizing weight
tensors in deep learning models to reduce inference latency and memory
footprint. TVM also has the ability to quantize weights and support low-bit
computations. Although quantization is typically expected to improve inference
time, in TVM, the performance of 8-bit quantization does not meet the
expectations. Typically, when applying 8-bit quantization to a deep learning
model, it is usually expected to achieve around 50% of the full-precision
inference time. However, in this particular case, not only does the quantized
version fail to achieve the desired performance boost, but it actually performs
worse, resulting in an inference time that is about 2 times as slow as the
non-quantized version. In this project, we thoroughly investigate the reasons
behind the underperformance and assess the compatibility and optimization
opportunities of 8-bit quantization in TVM. We discuss the optimization of two
different types of tasks: computation-bound and memory-bound, and provide a
detailed comparison of various optimization techniques in TVM. Through the
identification of performance issues, we have successfully improved
quantization by addressing a bug in graph building. Furthermore, we analyze
multiple optimization strategies to achieve the optimal quantization result.
The best experiment achieves 163.88% improvement compared with the TVM compiled
baseline in inference time for the compute-bound task and 194.98% for the
memory-bound task.

通过分析性能问题并解决图构建中的错误，我们成功改进了量化，并分析多种优化策略，实现了与 TVM 编译基准相比分别达到计算密集任务 163.88% 和内存密集任务 194.98% 的推理时间改进。

TVM 中的量化分析

Analyzing Quantization in TVM

While there exist many ways to deploy machine learning models on
microcontrollers, it is non-trivial to choose the optimal combination of
frameworks and targets for a given application. Thus, automating the end-to-end
benchmarking flow is of high relevance nowadays. A tool called MLonMCU is
proposed in this paper and demonstrated by benchmarking the state-of-the-art
TinyML frameworks TFLite for Microcontrollers and TVM effortlessly with a large
number of configurations in a low amount of time.

本文提出了一个名为 MLonMCU 的工具，通过在大量配置中自动化端到端基准测试流程并演示了 TFLite 和 TVM 等 TinyML 框架在微控制器上的无缝性能测试。

用快速重定向在微控制器上对 TinyML 进行基准测试

MLonMCU: TinyML Benchmarking with Fast Retargeting

There is an increasing need to bring machine learning to a wide diversity of
hardware devices. Current frameworks rely on vendor-specific operator libraries
and optimize for a narrow range of server-class GPUs. Deploying workloads to
new platforms -- such as mobile phones, embedded devices, and accelerators
(e.g., FPGAs, ASICs) -- requires significant manual effort. We propose TVM, a
compiler that exposes graph-level and operator-level optimizations to provide
performance portability to deep learning workloads across diverse hardware
back-ends. TVM solves optimization challenges specific to deep learning, such
as high-level operator fusion, mapping to arbitrary hardware primitives, and
memory latency hiding. It also automates optimization of low-level programs to
hardware characteristics by employing a novel, learning-based cost modeling
method for rapid exploration of code optimizations. Experimental results show
that TVM delivers performance across hardware back-ends that are competitive
with state-of-the-art, hand-tuned libraries for low-power CPU, mobile GPU, and
server-class GPUs. We also demonstrate TVM's ability to target new accelerator
back-ends, such as the FPGA-based generic deep learning accelerator. The system
is open sourced and in production use inside several major companies.

TVM 是一个深度学习编译器，可将深度学习工作负载在各种硬件后端上具有良好的性能移植性，通过学习基于成本模型的方法，自动进行低级程序的硬件特征优化，实验结果表明，TVM 在处理低功率 CPU、移动 GPU 和服务器级 GPU 的竞争性能与手工调优库类似，并能定位到新的加速器后端，如基于 FPGA 的深度学习通用加速器。