Large Language Models (LLMs) have played an important role in many fields due
to their powerful capabilities.However, their massive number of parameters
leads to high deployment requirements and incurs significant inference costs,
which impedes their practical applications. Training smaller models is an
effective way to address this problem. Therefore, we introduce OpenBA-V2, a
3.4B model derived from multi-stage compression and continual pre-training from
the original 15B OpenBA model. OpenBA-V2 utilizes more data, more flexible
training objectives, and techniques such as layer pruning, neural pruning, and
vocabulary pruning to achieve a compression rate of 77.3\% with minimal
performance loss. OpenBA-V2 demonstrates competitive performance compared to
other open-source models of similar size, achieving results close to or on par
with the 15B OpenBA model in downstream tasks such as common sense reasoning
and Named Entity Recognition (NER). OpenBA-V2 illustrates that LLMs can be
compressed into smaller ones with minimal performance loss by employing
advanced training objectives and data strategies, which may help deploy LLMs in
resource-limited scenarios.

通过采用先进的训练目标和数据策略，OpenBA-V2 以最小的性能损失将 LLMs 压缩为更小的模型，这在资源有限的场景中有助于部署 LLMs。

OpenBA-V2：使用快速多阶段剪枝达到 77.3% 的高压缩比

OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage  Pruning

Large transformer models have demonstrated remarkable success. Post-training
quantization (PTQ), which requires only a small dataset for calibration and
avoids end-to-end retraining, is a promising solution for compressing these
large models. Regrettably, existing PTQ methods typically exhibit non-trivial
performance loss. We find that the performance bottleneck stems from
over-consideration of hardware compatibility in the quantization process,
compelling them to reluctantly employ simple quantizers, albeit at the expense
of accuracy. With the above insights, we propose RepQuant, a novel PTQ
framework with quantization-inference decoupling paradigm to address the above
issues. RepQuant employs complex quantizers in the quantization process and
simplified quantizers in the inference process, and performs mathematically
equivalent transformations between the two through quantization scale
reparameterization, thus ensuring both accurate quantization and efficient
inference. More specifically, we focus on two components with extreme
distributions: LayerNorm activations and Softmax activations. Initially, we
apply channel-wise quantization and log$\sqrt{2}$ quantization, respectively,
which are tailored to their distributions. In particular, for the former, we
introduce a learnable per-channel dual clipping scheme, which is designed to
efficiently identify outliers in the unbalanced activations with fine
granularity. Then, we reparameterize the scales to hardware-friendly layer-wise
quantization and log2 quantization for inference. Moreover, quantized weight
reconstruction is seamlessly integrated into the above procedure to further
push the performance limits. Extensive experiments are performed on different
large-scale transformer variants on multiple tasks, including vision, language,
and multi-modal transformers, and RepQuant encouragingly demonstrates
significant performance advantages.

提出了一种新型的后训练量化框架 RepQuant，通过量化推理解耦合范式，采用复杂的量化器进行量化过程，采用简化的量化器进行推理过程，同时将量化标度重新参数化，以确保准确的量化和高效的推理，并通过集成量化权重重构进一步提高性能限制，经过广泛的实验验证在不同大规模变压器变体上在视觉、语言和多模态转换任务中取得了显著的性能优势。