Adapting pre-trained large language models to different domains in natural
language processing requires two key considerations: high computational demands
and model's inability to continual adaptation. To simultaneously address both
issues, this paper presents COPAL (COntinual Pruning in Adaptive Language
settings), an algorithm developed for pruning large language generative models
under a continual model adaptation setting. While avoiding resource-heavy
finetuning or retraining, our pruning process is guided by the proposed
sensitivity analysis. The sensitivity effectively measures model's ability to
withstand perturbations introduced by the new dataset and finds model's weights
that are relevant for all encountered datasets. As a result, COPAL allows
seamless model adaptation to new domains while enhancing the resource
efficiency. Our empirical evaluation on a various size of LLMs show that COPAL
outperforms baseline models, demonstrating its efficacy in efficiency and
adaptability.

本文提出了 COPAL 算法（COntinual Pruning in Adaptive Language settings）用于在持续的模型适应环境中对大型语言生成模型进行修剪，通过敏感性分析引导修剪过程，从而提高模型适应新领域的能力并增强资源效率。实证评估表明，COPAL 在效率和适应性方面优于基准模型。

COPAL：大型语言生成模型的持续剪枝

COPAL: Continual Pruning in Large Language Generative Models

In response to the rapidly evolving nature of adversarial attacks on a
monthly basis, numerous defenses have been proposed to generalize against as
many known attacks as possible. However, designing a defense method that can
generalize to all types of attacks, including unseen ones, is not realistic
because the environment in which defense systems operate is dynamic and
comprises various unique attacks used by many attackers. The defense system
needs to upgrade itself by utilizing few-shot defense feedback and efficient
memory. Therefore, we propose the first continual adversarial defense (CAD)
framework that adapts to any attacks in a dynamic scenario, where various
attacks emerge stage by stage. In practice, CAD is modeled under four
principles: (1) continual adaptation to new attacks without catastrophic
forgetting, (2) few-shot adaptation, (3) memory-efficient adaptation, and (4)
high accuracy on both clean and adversarial images. We leverage cutting-edge
continual learning, few-shot learning, and ensemble learning techniques to
qualify the principles. Experiments conducted on CIFAR-10 and ImageNet-100
validate the effectiveness of our approach against multiple stages of 10 modern
adversarial attacks and significant improvements over 10 baseline methods. In
particular, CAD is capable of quickly adapting with minimal feedback and a low
cost of defense failure, while maintaining good performance against old
attacks. Our research sheds light on a brand-new paradigm for continual defense
adaptation against dynamic and evolving attacks.

提出首个连续性对抗防御（CAD）框架，适应动态场景中的任何攻击，并通过实验证明其对多个现代对抗攻击的有效性和对 10 个基线方法的显著改进。

持续对抗性防御

Continual Adversarial Defense

Domain adaptation (DA) strives to mitigate the domain gap between the source
domain where a model is trained, and the target domain where the model is
deployed. When a deep learning model is deployed on an aerial platform, it may
face gradually degrading weather conditions during operation, leading to
widening domain gaps between the training data and the encountered evaluation
data. We synthesize two such gradually worsening weather conditions on real
images from two existing aerial imagery datasets, generating a total of four
benchmark datasets. Under the continual, or test-time adaptation setting, we
evaluate three DA models on our datasets: a baseline standard DA model and two
continual DA models. In such setting, the models can access only one small
portion, or one batch of the target data at a time, and adaptation takes place
continually, and over only one epoch of the data. The combination of the
constraints of continual adaptation, and gradually deteriorating weather
conditions provide the practical DA scenario for aerial deployment. Among the
evaluated models, we consider both convolutional and transformer architectures
for comparison. We discover stability issues during adaptation for existing
buffer-fed continual DA methods, and offer gradient normalization as a simple
solution to curb training instability.

领域适应旨在减少深度学习模型训练所在源领域和部署所在目标领域之间的差距。在航空平台上部署的深度学习模型可能面临逐渐恶化的天气条件，导致训练数据和评估数据之间的领域差距扩大。我们合成了两种逐渐恶化的天气条件并应用于真实图像数据集，生成了四个基准数据集。在持续或测试时间适应设置下，我们评估了三种领域适应模型：一个基准标准领域适应模型和两个持续领域适应模型。在这种设置下，模型一次只能访问目标数据中的一小部分或一个批次，并且适应过程持续进行，仅在数据的一个时代内完成。持续适应和逐渐恶劣的天气条件的组合为航空部署提供了实际的领域适应场景。在评估的模型中，我们考虑卷积和变换器体系结构进行比较。我们发现现有的基于缓冲区的持续领域适应方法在适应过程中存在稳定性问题，并提出梯度归一化作为一种简单的解决方案以抑制训练不稳定性。

逐渐恶化天气下的航空影像持续领域适应

Continual Domain Adaptation on Aerial Images under Gradually Degrading  Weather

Depth estimation from stereo images is carried out with unmatched results by
convolutional neural networks trained end-to-end to regress dense disparities.
Like for most tasks, this is possible if large amounts of labelled samples are
available for training, possibly covering the whole data distribution
encountered at deployment time. Being such an assumption systematically unmet
in real applications, the capacity of adapting to any unseen setting becomes of
paramount importance. Purposely, we propose a continual adaptation paradigm for
deep stereo networks designed to deal with challenging and ever-changing
environments. We design a lightweight and modular architecture, Modularly
ADaptive Network (MADNet), and formulate Modular ADaptation algorithms (MAD,
MAD++) which permit efficient optimization of independent sub-portions of the
entire network. In our paradigm, the learning signals needed to continuously
adapt models online can be sourced from self-supervision via right-to-left
image warping or from traditional stereo algorithms. With both sources, no
other data than the input images being gathered at deployment time are needed.
Thus, our network architecture and adaptation algorithms realize the first
real-time self-adaptive deep stereo system and pave the way for a new paradigm
that can facilitate practical deployment of end-to-end architectures for dense
disparity regression.

提出一种 Modularly ADaptive Network (MADNet) 架构和 Modular ADaptation (MAD, MAD ++) 算法设计，实现在不同环境下对深度估计进行实时自适应的端到端架构，无需与输入图像不同的数据源。