Large language models (LLMs) show inherent brittleness in their safety
mechanisms, as evidenced by their susceptibility to jailbreaking and even
non-malicious fine-tuning. This study explores this brittleness of safety
alignment by leveraging pruning and low-rank modifications. We develop methods
to identify critical regions that are vital for safety guardrails, and that are
disentangled from utility-relevant regions at both the neuron and rank levels.
Surprisingly, the isolated regions we find are sparse, comprising about $3\%$
at the parameter level and $2.5\%$ at the rank level. Removing these regions
compromises safety without significantly impacting utility, corroborating the
inherent brittleness of the model's safety mechanisms. Moreover, we show that
LLMs remain vulnerable to low-cost fine-tuning attacks even when modifications
to the safety-critical regions are restricted. These findings underscore the
urgent need for more robust safety strategies in LLMs.

利用剪枝和低秩修改探索大型语言模型的安全与稳健性，发现关键区域的剔除威胁了安全性但对效用影响不大，同时指出即使限制对关键区域的修改，大型语言模型仍然容易受到低成本的微调攻击，强调了对大型语言模型更强健的安全策略的迫切需求。

通过修剪和低秩修改评估安全对齐的脆弱性

Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank  Modifications

Large Language Models (LLMs) have recently gained popularity due to their
impressive few-shot performance across various downstream tasks. However,
fine-tuning all parameters and storing a unique model for each downstream task
or domain becomes impractical because of the massive size of checkpoints (e.g.,
350GB in GPT-3). Current literature, such as LoRA, showcases the potential of
low-rank modifications to the original weights of an LLM, enabling efficient
adaptation and storage for task-specific models. These methods can reduce the
number of parameters needed to fine-tune an LLM by several orders of magnitude.
Yet, these methods face two primary limitations: 1) the parameter reduction is
lower-bounded by the rank one decomposition, and 2) the extent of reduction is
heavily influenced by both the model architecture and the chosen rank. For
instance, in larger models, even a rank one decomposition might exceed the
number of parameters truly needed for adaptation. In this paper, we introduce
NOLA, which overcomes the rank one lower bound present in LoRA. It achieves
this by re-parameterizing the low-rank matrices in LoRA using linear
combinations of randomly generated matrices (basis) and optimizing the linear
mixture coefficients only. This approach allows us to decouple the number of
trainable parameters from both the choice of rank and the network architecture.
We present adaptation results using GPT-2 and ViT in natural language and
computer vision tasks. NOLA performs as well as, or better than models with
equivalent parameter counts. Furthermore, we demonstrate that we can halve the
parameters in larger models compared to LoRA with rank one, without sacrificing
performance.

通过重新参数化使用线性组合随机生成的矩阵（基）的低秩矩阵，并仅优化线性混合系数，NOLA 将创新地克服了 LoRA 中的秩一下界，与等效参数计数的模型相比，NOLA 在自然语言处理和计算机视觉任务中表现得很好，并且证明能在更大的模型中减少一半的参数而不降低性能。