Tensor processing units (TPUs), specialized hardware accelerators for machine
learning tasks, have shown significant performance improvements when executing
convolutional layers in convolutional neural networks (CNNs). However, they
struggle to maintain the same efficiency in fully connected (FC) layers,
leading to suboptimal hardware utilization. In-memory analog computing (IMAC)
architectures, on the other hand, have demonstrated notable speedup in
executing FC layers. This paper introduces a novel, heterogeneous,
mixed-signal, and mixed-precision architecture that integrates an IMAC unit
with an edge TPU to enhance mobile CNN performance. To leverage the strengths
of TPUs for convolutional layers and IMAC circuits for dense layers, we propose
a unified learning algorithm that incorporates mixed-precision training
techniques to mitigate potential accuracy drops when deploying models on the
TPU-IMAC architecture. The simulations demonstrate that the TPU-IMAC
configuration achieves up to $2.59\times$ performance improvements, and $88\%$
memory reductions compared to conventional TPU architectures for various CNN
models while maintaining comparable accuracy. The TPU-IMAC architecture shows
potential for various applications where energy efficiency and high performance
are essential, such as edge computing and real-time processing in mobile
devices. The unified training algorithm and the integration of IMAC and TPU
architectures contribute to the potential impact of this research on the
broader machine learning landscape.

提出了一种新颖的异构混合信号混合精度架构，将 IMAC 单元与边缘 TPU 集成以提高移动 CNN 性能，借助 TPUs 卷积层和 IMAC 电路密集层的优势，设计了一种统一的混合精度训练算法，模拟结果表明，TPU-IMAC 构型在各种 CNN 模型中实现了高达 2.59 倍的性能提升和 88％的内存减少，同时保持了可比较的准确性。该架构对于边缘计算和移动设备中的实时处理等需要能源效率和高性能的应用具有潜在的影响。

内存模拟计算架构与张量处理单元的异构集成

Heterogeneous Integration of In-Memory Analog Computing Architectures  with Tensor Processing Units

The latest industrial inference engines, such as FasterTransformer1 and
TurboTransformers, have verified that half-precision floating point (FP16) and
8-bit integer (INT8) quantization can greatly improve model inference speed.
However, the existing FP16 or INT8 quantization methods are too complicated,
and improper usage will lead to performance damage greatly. In this paper, we
develop a toolkit for users to easily quantize their models for inference, in
which a Self-Adaptive Mixed-Precision (SAMP) is proposed to automatically
control quantization rate by a mixed-precision architecture to balance
efficiency and performance. Experimental results show that our SAMP toolkit has
a higher speedup than PyTorch and FasterTransformer while ensuring the required
performance. In addition, SAMP is based on a modular design, decoupling the
tokenizer, embedding, encoder and target layers, which allows users to handle
various downstream tasks and can be seamlessly integrated into PyTorch.

本文提出了一种自适应混合精度 (SAMP) 工具包来控制量化率，以平衡效率和性能，可以轻松量化模型进行推理。实验结果表明，SAMP 工具包的加速比优于 PyTorch 和 FasterTransformer，同时保证所需性能。此外，SAMP 基于模块化设计，可无缝集成到 PyTorch 中处理各种下游任务。