Large Language Models (LLMs) have played an important role in many fields due
to their powerful capabilities.However, their massive number of parameters
leads to high deployment requirements and incurs significant inference costs,
which impedes their practical applications. Training smaller models is an
effective way to address this problem. Therefore, we introduce OpenBA-V2, a
3.4B model derived from multi-stage compression and continual pre-training from
the original 15B OpenBA model. OpenBA-V2 utilizes more data, more flexible
training objectives, and techniques such as layer pruning, neural pruning, and
vocabulary pruning to achieve a compression rate of 77.3\% with minimal
performance loss. OpenBA-V2 demonstrates competitive performance compared to
other open-source models of similar size, achieving results close to or on par
with the 15B OpenBA model in downstream tasks such as common sense reasoning
and Named Entity Recognition (NER). OpenBA-V2 illustrates that LLMs can be
compressed into smaller ones with minimal performance loss by employing
advanced training objectives and data strategies, which may help deploy LLMs in
resource-limited scenarios.

通过采用先进的训练目标和数据策略，OpenBA-V2 以最小的性能损失将 LLMs 压缩为更小的模型，这在资源有限的场景中有助于部署 LLMs。

OpenBA-V2：使用快速多阶段剪枝达到 77.3% 的高压缩比

OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage  Pruning

In this study, we explore an innovative approach for neural network
optimization, focusing on the application of gradient sampling techniques,
similar to those in StochGradAdam, during the pruning process. Our primary
objective is to maintain high accuracy levels in pruned models, a critical
challenge in resource-limited scenarios. Our extensive experiments reveal that
models optimized with gradient sampling techniques are more effective at
preserving accuracy during pruning compared to those using traditional
optimization methods. This finding underscores the significance of gradient
sampling in facilitating robust learning and enabling networks to retain
crucial information even after substantial reduction in their complexity. We
validate our approach across various datasets and neural architectures,
demonstrating its broad applicability and effectiveness. The paper also delves
into the theoretical aspects, explaining how gradient sampling techniques
contribute to the robustness of models during pruning. Our results suggest a
promising direction for creating efficient neural networks that do not
compromise on accuracy, even in environments with constrained computational
resources.

本研究探讨了一种创新的神经网络优化方法，专注于在修剪过程中应用与 StochGradAdam 类似的梯度采样技术。我们的主要目标是在修剪模型时保持高精度水平，这是资源受限场景中的一个重要挑战。我们广泛的实验证明，使用梯度采样技术优化的模型在修剪过程中相对于使用传统优化方法的模型更有效地保持准确性。这一发现强调了梯度采样在促进稳健学习方面的重要性，使得网络即使在复杂度大幅减少后仍能保留关键信息。我们在各种数据集和神经网络结构上验证了我们的方法，展示了其广泛的适用性和有效性。该论文还深入研究了理论方面，解释了梯度采样技术在修剪过程中对模型稳健性的贡献。我们的结果为在计算资源受限的环境中创建不会牺牲准确性的高效神经网络指明了一个有希望的方向。

具有梯度采样优化的残差神经网络的鲁棒剪枝

Robust Neural Pruning with Gradient Sampling Optimization for Residual  Neural Networks

With the development of large pre-trained vision-language models, how to
effectively transfer the knowledge of such foundational models to downstream
tasks becomes a hot topic, especially in a data-deficient scenario. Recently,
prompt tuning has become a popular solution. When adapting the vision-language
models, researchers freeze the parameters in the backbone and only design and
tune the prompts. On the one hand, the delicate design of prompt tuning
exhibits strong performance. On the other hand, complicated structures and
update rules largely increase the computation and storage cost. Motivated by
the observation that the evolution pattern of the generalization capability in
visual-language models aligns harmoniously with the trend of rank variations in
the prompt matrix during adaptation, we design a new type of prompt,
Re-parameterized Low-rank Prompt (RLP), for both efficient and effective
adaptation. Our method could largely reduce the number of tunable parameters
and storage space, which is quite beneficial in resource-limited scenarios.
Extensive experiments further demonstrate the superiority of RLP. In
particular, RLP shows comparable or even stronger performance than the latest
state-of-the-art methods with an extremely small number of parameters. On a
series of tasks over 11 datasets, RLP significantly increases the average
downstream accuracy of classic prompt tuning by up to 5.25% using merely 0.5K
parameters.

通过重新参数化低秩提示（RLP），我们设计了一种新型提示，用于高效和有效地适应基于视觉和语言的模型，并显著增加了传统提示调整的平均下游准确性，仅使用 0.5K 的参数。

重新参数化的低秩提示：在 0.5K 参数范围内泛化视觉语言模型

Re-parameterized Low-rank Prompt: Generalize a Vision-Language Model  within 0.5K Parameters

Active speaker detection is a challenging task in audio-visual scenario
understanding, which aims to detect who is speaking in one or more speakers
scenarios. This task has received extensive attention as it is crucial in
applications such as speaker diarization, speaker tracking, and automatic video
editing. The existing studies try to improve performance by inputting multiple
candidate information and designing complex models. Although these methods
achieved outstanding performance, their high consumption of memory and
computational power make them difficult to be applied in resource-limited
scenarios. Therefore, we construct a lightweight active speaker detection
architecture by reducing input candidates, splitting 2D and 3D convolutions for
audio-visual feature extraction, and applying gated recurrent unit (GRU) with
low computational complexity for cross-modal modeling. Experimental results on
the AVA-ActiveSpeaker dataset show that our framework achieves competitive mAP
performance (94.1% vs. 94.2%), while the resource costs are significantly lower
than the state-of-the-art method, especially in model parameters (1.0M vs.
22.5M, about 23x) and FLOPs (0.6G vs. 2.6G, about 4x). In addition, our
framework also performs well on the Columbia dataset showing good robustness.
The code and model weights are available at
this https URL

提出了一种轻量级的主动说话人检测框架，通过减少输入候选信息、分离 2D 和 3D 卷积进行音视频特征提取、并应用门控循环单元（GRU）进行跨模态建模，可以在计算、资源消耗上比现有方法表现更优。