Vision Transformers (ViTs) have achieved remarkable performance in various
image classification tasks by leveraging the attention mechanism to process
image patches as tokens. However, the high computational and memory demands of
ViTs pose significant challenges for deployment in resource-constrained
environments. This paper introduces ViT-1.58b, a novel 1.58-bit quantized ViT
model designed to drastically reduce memory and computational overhead while
preserving competitive performance. ViT-1.58b employs ternary quantization,
which refines the balance between efficiency and accuracy by constraining
weights to {-1, 0, 1} and quantizing activations to 8-bit precision. Our
approach ensures efficient scaling in terms of both memory and computation.
Experiments on CIFAR-10 and ImageNet-1k demonstrate that ViT-1.58b maintains
comparable accuracy to full-precision Vit, with significant reductions in
memory usage and computational costs. This paper highlights the potential of
extreme quantization techniques in developing sustainable AI solutions and
contributes to the broader discourse on efficient model deployment in practical
applications. Our code and weights are available at
this https URL

ViT-1.58b 是一种新的 1.58-bit 量化的视觉 Transformer 模型，通过三值量化的方式来平衡效率和准确性，实现了在存储和计算开销方面的高效扩展。实验证明，ViT-1.58b 与全精度 ViT 相比，在减少存储使用和计算成本方面取得了显著的成果，从而突出了极端量化技术在开发可持续的人工智能解决方案方面的潜力，并为实际应用中高效模型部署的广泛讨论做出了贡献。

ViT-1.58b：1-bit 时代的移动视觉 Transformer

ViT-1.58b: Mobile Vision Transformers in the 1-bit Era

Inference time, model size, and accuracy are critical for deploying deep
neural network models. Numerous research efforts have been made to compress
neural network models with faster inference and higher accuracy. Pruning and
quantization are mainstream methods to this end. During model quantization,
converting individual float values of layer weights to low-precision ones can
substantially reduce the computational overhead and improve the inference
speed. Many quantization methods have been studied, for example, vector
quantization, low-bit quantization, and binary/ternary quantization. This
survey focuses on ternary quantization. We review the evolution of ternary
quantization and investigate the relationships among existing ternary
quantization methods from the perspective of projection function and
optimization methods.

本研究综述了深度神经网络模型压缩方法中的一种，三值量化。研究回顾了三值量化的发展历程，并从投影函数和优化方法的角度研究了现有的三值量化方法之间的关系。