Zeroth-order optimization (ZO) is a memory-efficient strategy for fine-tuning
Large Language Models using only forward passes. However, the application of ZO
fine-tuning in memory-constrained settings such as mobile phones and laptops is
still challenging since full precision forward passes are infeasible. In this
study, we address this limitation by integrating sparsity and quantization into
ZO fine-tuning of LLMs. Specifically, we investigate the feasibility of
fine-tuning an extremely small subset of LLM parameters using ZO. This approach
allows the majority of un-tuned parameters to be quantized to accommodate the
constraint of limited device memory. Our findings reveal that the pre-training
process can identify a set of "sensitive parameters" that can guide the ZO
fine-tuning of LLMs on downstream tasks. Our results demonstrate that
fine-tuning 0.1% sensitive parameters in the LLM with ZO can outperform the
full ZO fine-tuning performance, while offering wall-clock time speedup.
Additionally, we show that ZO fine-tuning targeting these 0.1% sensitive
parameters, combined with 4 bit quantization, enables efficient ZO fine-tuning
of an Llama2-7B model on a GPU device with less than 8 GiB of memory and
notably reduced latency.

本研究通过将稀疏性和量化技术整合到零阶优化（ZO）细调的大型语言模型（LLM）中，从而解决在内存受限环境（如移动电话和笔记本电脑）中使用 ZO 细调的挑战。研究结果表明，使用 ZO 对 LLM 进行 0.1% 敏感参数细调能优于全面细调，并同时提供加速的速度。此外，结合 4 位量化技术，ZO 对 Llama2-7B 模型的高效细调在 GPU 设备上不到 8 GB 内存的限制下实现了显著降低的延迟。

零阶极度稀疏 LLMs 的微调

Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity

Continual learning is a sub-field of machine learning, which aims to allow
machine learning models to continuously learn on new data, by accumulating
knowledge without forgetting what was learned in the past. In this work, we
take a step back, and ask: "Why should one care about continual learning in the
first place?". We set the stage by surveying recent continual learning papers
published at three major machine learning conferences, and show that
memory-constrained settings dominate the field. Then, we discuss five open
problems in machine learning, and even though they seem unrelated to continual
learning at first sight, we show that continual learning will inevitably be
part of their solution. These problems are model-editing, personalization,
on-device learning, faster (re-)training and reinforcement learning. Finally,
by comparing the desiderata from these unsolved problems and the current
assumptions in continual learning, we highlight and discuss four future
directions for continual learning research. We hope that this work offers an
interesting perspective on the future of continual learning, while displaying
its potential value and the paths we have to pursue in order to make it
successful. This work is the result of the many discussions the authors had at
the Dagstuhl seminar on Deep Continual Learning, in March 2023.

连续学习是机器学习的一个子领域，本文调查了连续学习研究中的内存受限设置、未解决的问题以及未来的研究方向。

持续学习：应用与未来发展

Continual Learning: Applications and the Road Forward

Autocomplete is a task where the user inputs a piece of text, termed prompt,
which is conditioned by the model to generate semantically coherent
continuation. Existing works for this task have primarily focused on datasets
(e.g., email, chat) with high frequency user prompt patterns (or focused
prompts) where word-based language models have been quite effective. In this
work, we study the more challenging setting consisting of low frequency user
prompt patterns (or broad prompts, e.g., prompt about 93rd academy awards) and
demonstrate the effectiveness of character-based language models. We study this
problem under memory-constrained settings (e.g., edge devices and smartphones),
where character-based representation is effective in reducing the overall model
size (in terms of parameters). We use WikiText-103 benchmark to simulate broad
prompts and demonstrate that character models rival word models in exact match
accuracy for the autocomplete task, when controlled for the model size. For
instance, we show that a 20M parameter character model performs similar to an
80M parameter word model in the vanilla setting. We further propose novel
methods to improve character models by incorporating inductive bias in the form
of compositional information and representation transfer from large word
models.

本文研究的是低频用户提示模式或广泛提示的自动完成任务以及在受限内存下使用基于字符的语言模型降低整体模型大小的效果。研究证明，当控制模型大小时，字符模型的自动完成任务的精确匹配准确性与字模型相当。作者进一步尝试在字符模型中融入大型字模型的组成信息和表现转移的归纳偏差。