Large language models (LLMs) have recently emerged as powerful tools for
tackling many language-processing tasks. Despite their success, training and
fine-tuning these models is still far too computationally and memory intensive.
In this paper, we identify and characterise the important components needed for
effective model convergence using gradient descent. In doing so we find that
the intermediate activations used to implement backpropagation can be
excessively compressed without incurring any degradation in performance. This
result leads us to a cheap and memory-efficient algorithm for both fine-tuning
and pre-training LLMs. The proposed algorithm simply divides the tokens up into
smaller sub-tokens before projecting them onto a fixed 1-dimensional subspace
during the forward pass. These features are then coarsely reconstructed during
the backward pass to implement the update rules. We confirm the effectiveness
of our algorithm as being complimentary to many state-of-the-art PEFT methods
on the VTAB-1k fine-tuning benchmark. Furthermore, we outperform QLoRA for
fine-tuning LLaMA and show competitive performance against other
memory-efficient pre-training methods on the large-scale C4 dataset.

通过梯度下降，我们确认了大型语言模型的有效模型收敛所需的重要组件，并提出了一种廉价且内存高效的算法来进行微调和预训练大型语言模型。

VeLoRA：使用排序 - 1 子标记投影的内存高效训练

VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections

Achieving sample efficiency in online episodic reinforcement learning (RL)
requires optimally balancing exploration and exploitation. When it comes to a
finite-horizon episodic Markov decision process with $S$ states, $A$ actions
and horizon length $H$, substantial progress has been achieved towards
characterizing the minimax-optimal regret, which scales on the order of
$\sqrt{H^2SAT}$ (modulo log factors) with $T$ the total number of samples.
While several competing solution paradigms have been proposed to minimize
regret, they are either memory-inefficient, or fall short of optimality unless
the sample size exceeds an enormous threshold (e.g., $S^6A^4
\,\mathrm{poly}(H)$ for existing model-free methods).
To overcome such a large sample size barrier to efficient RL, we design a
novel model-free algorithm, with space complexity $O(SAH)$, that achieves
near-optimal regret as soon as the sample size exceeds the order of
$SA\,\mathrm{poly}(H)$. In terms of this sample size requirement (also referred
to the initial burn-in cost), our method improves -- by at least a factor of
$S^5A^3$ -- upon any prior memory-efficient algorithm that is asymptotically
regret-optimal. Leveraging the recently introduced variance reduction strategy
(also called {\em reference-advantage decomposition}), the proposed algorithm
employs an {\em early-settled} reference update rule, with the aid of two
Q-learning sequences with upper and lower confidence bounds. The design
principle of our early-settled variance reduction method might be of
independent interest to other RL settings that involve intricate
exploration-exploitation trade-offs.

通过引入方差缩减策略，设计了一个记忆高效的算法来解决在线序列化强化学习中的勘探和开发之间的平衡问题，该算法的空间复杂度为 $ O (SAH)$，较以前的算法提高了 $S^5A^3$ 倍的效率。

突破样本复杂度障碍，实现后悔优化的无模型强化学习

Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free  Reinforcement Learning

In applications such as natural language processing or computer vision, one
is given a large $n \times d$ matrix $A = (a_{i,j})$ and would like to compute
a matrix decomposition, e.g., a low rank approximation, of a function $f(A) =
(f(a_{i,j}))$ applied entrywise to $A$. A very important special case is the
likelihood function $f\left( A \right ) = \log{\left( \left| a_{ij}\right|
+1\right)}$. A natural way to do this would be to simply apply $f$ to each
entry of $A$, and then compute the matrix decomposition, but this requires
storing all of $A$ as well as multiple passes over its entries. Recent work of
Liang et al.\ shows how to find a rank-$k$ factorization to $f(A)$ for an $n
\times n$ matrix $A$ using only $n \cdot \operatorname{poly}(\epsilon^{-1}k\log
n)$ words of memory, with overall error $10\|f(A)-[f(A)]_k\|_F^2 +
\operatorname{poly}(\epsilon/k) \|f(A)\|_{1,2}^2$, where $[f(A)]_k$ is the best
rank-$k$ approximation to $f(A)$ and $\|f(A)\|_{1,2}^2$ is the square of the
sum of Euclidean lengths of rows of $f(A)$. Their algorithm uses three passes
over the entries of $A$. The authors pose the open question of obtaining an
algorithm with $n \cdot \operatorname{poly}(\epsilon^{-1}k\log n)$ words of
memory using only a single pass over the entries of $A$. In this paper we
resolve this open question, obtaining the first single-pass algorithm for this
problem and for the same class of functions $f$ studied by Liang et al.
Moreover, our error is $\|f(A)-[f(A)]_k\|_F^2 + \operatorname{poly}(\epsilon/k)
\|f(A)\|_F^2$, where $\|f(A)\|_F^2$ is the sum of squares of Euclidean lengths
of rows of $f(A)$. Thus our error is significantly smaller, as it removes the
factor of $10$ and also $\|f(A)\|_F^2 \leq \|f(A)\|_{1,2}^2$. We also give an
algorithm for regression, pointing out an error in previous work, and
empirically validate our results.

本文提出了一种用于矩阵分解且只需一次通过矩阵的内部即可完成的内存高效算法，并且在同样类似于 Liang et al. 所研究的函数 $f$ 的条件下，误差显著减小，还提出了应用于回归问题的算法，并通过实验证明了其结果的正确性。