Ternary neural networks (TNNs) offer a superior accuracy-energy trade-off
compared to binary neural networks. However, until now, they have required
specialized accelerators to realize their efficiency potential, which has
hindered widespread adoption. To address this, we present xTern, a lightweight
extension of the RISC-V instruction set architecture (ISA) targeted at
accelerating TNN inference on general-purpose cores. To complement the ISA
extension, we developed a set of optimized kernels leveraging xTern, achieving
67% higher throughput than their 2-bit equivalents. Power consumption is only
marginally increased by 5.2%, resulting in an energy efficiency improvement by
57.1%. We demonstrate that the proposed xTern extension, integrated into an
octa-core compute cluster, incurs a minimal silicon area overhead of 0.9% with
no impact on timing. In end-to-end benchmarks, we demonstrate that xTern
enables the deployment of TNNs achieving up to 1.6 percentage points higher
CIFAR-10 classification accuracy than 2-bit networks at equal inference
latency. Our results show that xTern enables RISC-V-based ultra-low-power edge
AI platforms to benefit from the efficiency potential of TNNs.

Ternary neural networks offer superior accuracy-energy trade-off, and the xTern extension of the RISC-V instruction set architecture accelerates TNN inference on general-purpose cores, achieving higher throughput, improved energy efficiency, and higher CIFAR-10 classification accuracy.

xTern: 在基于 RISC-V 的边缘系统上的高能效三值神经网络推理

xTern: Energy-Efficient Ternary Neural Network Inference on RISC-V-Based  Edge Systems

Ternary and binary neural networks enable multiplication-free computation and
promise multiple orders of magnitude efficiency gains over full-precision
networks if implemented on specialized hardware. However, since both the
parameter and the output space are highly discretized, such networks have
proven very difficult to optimize. The difficulties are compounded for the
class of transformer text generation models due to the sensitivity of the
attention operation to quantization and the noise-compounding effects of
autoregressive decoding in the high-cardinality output space. We approach the
problem with a mix of statistics-based quantization for the weights and elastic
quantization of the activations and demonstrate the first ternary and binary
transformer models on the downstream tasks of summarization and machine
translation. Our ternary BART base achieves an R1 score of 41 on the
CNN/DailyMail benchmark, which is merely 3.9 points behind the full model while
being 16x more efficient. Our binary model, while less accurate, achieves a
highly non-trivial score of 35.6. For machine translation, we achieved BLEU
scores of 21.7 and 17.6 on the WMT16 En-Ro benchmark, compared with a full
precision mBART model score of 26.8. We also compare our approach in the 8-bit
activation setting, where our ternary and even binary weight models can match
or outperform the best existing 8-bit weight models in the literature. Our code
and models are available at:
this https URL

采用一种基于统计的权重量化和弹性激活量化的方法实现了第一批三元和双元变压器模型，其在摘要和机器翻译的下游任务中达到了较高的得分，同时比全精度模型高效 16 倍以上。

二元和三元自然语言生成

Binary and Ternary Natural Language Generation

Ternary Neural Networks (TNNs) have received much attention due to being
potentially orders of magnitude faster in inference, as well as more power
efficient, than full-precision counterparts. However, 2 bits are required to
encode the ternary representation with only 3 quantization levels leveraged. As
a result, conventional TNNs have similar memory consumption and speed compared
with the standard 2-bit models, but have worse representational capability.
Moreover, there is still a significant gap in accuracy between TNNs and
full-precision networks, hampering their deployment to real applications. To
tackle these two challenges, in this work, we first show that, under some mild
constraints, computational complexity of the ternary inner product can be
reduced by a factor of 2. Second, to mitigate the performance gap, we
elaborately design an implementation-dependent ternary quantization algorithm.
The proposed framework is termed Fast and Accurate Ternary Neural Networks
(FATNN). Experiments on image classification demonstrate that our FATNN
surpasses the state-of-the-arts by a significant margin in accuracy. More
importantly, speedup evaluation compared with various precisions is analyzed on
several platforms, which serves as a strong benchmark for further research.

本文提出了一种快速高效的三值神经网络 (FATNN) 框架，通过精心设计实现依存的三值量化算法，针对计算复杂度和性能差异等挑战，大幅度提高了图像分类准确性，并在多个平台上进行了速度评估，成为进一步研究的强大基准。

FATNN: 快速和准确的三元神经网络

FATNN: Fast and Accurate Ternary Neural Networks

The computation and storage requirements for Deep Neural Networks (DNNs) are
usually high. This issue limits their deployability on ubiquitous computing
devices such as smart phones, wearables and autonomous drones. In this paper,
we propose ternary neural networks (TNNs) in order to make deep learning more
resource-efficient. We train these TNNs using a teacher-student approach based
on a novel, layer-wise greedy methodology. Thanks to our two-stage training
procedure, the teacher network is still able to use state-of-the-art methods
such as dropout and batch normalization to increase accuracy and reduce
training time. Using only ternary weights and activations, the student ternary
network learns to mimic the behavior of its teacher network without using any
multiplication. Unlike its -1,1 binary counterparts, a ternary neural network
inherently prunes the smaller weights by setting them to zero during training.
This makes them sparser and thus more energy-efficient. We design a
purpose-built hardware architecture for TNNs and implement it on FPGA and ASIC.
We evaluate TNNs on several benchmark datasets and demonstrate up to 3.1x
better energy efficiency with respect to the state of the art while also
improving accuracy.

本文提出了基于三值神经网络和师生模型的新型深度学习方法，并在 FPGA 和 ASIC 硬件上实现；相比现有技术，该方案在提高准确率的前提下，节约了高达 3.1 倍的能源。