With the remarkable capabilities, large language models (LLMs) have emerged
as essential elements in numerous NLP applications, while parameter-efficient
finetuning, especially LoRA, has gained popularity as a lightweight approach
for model customization. Meanwhile, various dropout methods, initially designed
for full finetuning with all the parameters updated, alleviates overfitting
associated with excessive parameter redundancy. Hence, a possible contradiction
arises from negligible trainable parameters of LoRA and the effectiveness of
previous dropout methods, which has been largely overlooked. To fill this gap,
we first confirm that parameter-efficient LoRA is also overfitting-prone. We
then revisit transformer-specific dropout methods, and establish their
equivalence and distinctions mathematically and empirically. Building upon this
comparative analysis, we introduce a unified framework for a comprehensive
investigation, which instantiates these methods based on dropping position,
structural pattern and compensation measure. Through this framework, we reveal
the new preferences and performance comparisons of them when involved with
limited trainable parameters. This framework also allows us to amalgamate the
most favorable aspects into a novel dropout method named HiddenKey. Extensive
experiments verify the remarkable superiority and sufficiency of HiddenKey
across multiple models and tasks, which highlights it as the preferred approach
for high-performance and parameter-efficient finetuning of LLMs.

基于对参数高效 LoRA 的研究，我们重新审视了特定于 Transformer 的 Dropout 方法的数学和经验上的等价性和区别，并基于此提出了一个统一的框架，揭示了当涉及到有限可训练参数时它们的新偏好和性能比较。这个框架还允许我们将最有利的方面融合成一个名为 HiddenKey 的新的 Dropout 方法，广泛的实验证实 HiddenKey 在多个模型和任务上具有显著的优势和足够性，将其作为大语言模型高性能和参数高效微调的首选方法。

LoRA 遇见 Dropout: 一个统一框架下的研究

LoRA Meets Dropout under a Unified Framework

In encoder-decoder neural models, multiple encoders are in general used to
represent the contextual information in addition to the individual sentence. In
this paper, we investigate multi-encoder approaches in documentlevel neural
machine translation (NMT). Surprisingly, we find that the context encoder does
not only encode the surrounding sentences but also behaves as a noise
generator. This makes us rethink the real benefits of multi-encoder in
context-aware translation - some of the improvements come from robust training.
We compare several methods that introduce noise and/or well-tuned dropout setup
into the training of these encoders. Experimental results show that noisy
training plays an important role in multi-encoder-based NMT, especially when
the training data is small. Also, we establish a new state-of-the-art on IWSLT
Fr-En task by careful use of noise generation and dropout methods.

研究多编码器方法在文档级神经机器翻译中的应用，证实上下文编码器不仅编码周围的句子还会产生噪声，探讨引入噪音和严谨的 dropout 方法在多编码器 NMT 中的作用，实验结果证明噪声训练在小数据集中发挥重要作用，并通过使用噪音生成和 dropout 方法，在 IWSLT Fr-En 任务上创立了新的最优效果。

多编码器有帮助吗？基于上下文的神经机器翻译案例研究

Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine  Translation

Dropout methods are a family of stochastic techniques used in neural network
training or inference that have generated significant research interest and are
widely used in practice. They have been successfully applied in neural network
regularization, model compression, and in measuring the uncertainty of neural
network outputs. While original formulated for dense neural network layers,
recent advances have made dropout methods also applicable to convolutional and
recurrent neural network layers. This paper summarizes the history of dropout
methods, their various applications, and current areas of research interest.
Important proposed methods are described in additional detail.

本文总结了 dropout 方法的历史、各种应用以及当前研究兴趣的领域，同时详细描述了一些重要的提出方法。