Large language models (LLMs) are capable of performing conditional sequence
generation tasks, such as translation or summarization, through instruction
fine-tuning. The fine-tuning data is generally sequentially concatenated from a
specific task instruction, an input sentence, and the corresponding response.
Considering the locality modeled by the self-attention mechanism of LLMs, these
models face the risk of instruction forgetting when generating responses for
long input sentences. To mitigate this issue, we propose enhancing the
instruction-following capability of LLMs by shifting the position of task
instructions after the input sentences. Theoretical analysis suggests that our
straightforward method can alter the model's learning focus, thereby
emphasizing the training of instruction-following capabilities. Concurrently,
experimental results demonstrate that our approach consistently outperforms
traditional settings across various model scales (1B / 7B / 13B) and different
sequence generation tasks (translation and summarization), without any
additional data or annotation costs. Notably, our method significantly improves
the zero-shot performance on conditional sequence generation, e.g., up to 9.7
BLEU points on WMT zero-shot translation tasks.

通过改变任务指令在输入句子之后的位置，我们提出了一种增强大型语言模型的指令遵循功能的方法，该方法可以显著改善条件序列生成的零样本性能。

大型语言模型中的指导位置在序列生成中的作用

Instruction Position Matters in Sequence Generation with Large Language  Models

There has been an increased interest in the integration of pretrained speech
recognition (ASR) and language models (LM) into the SLU framework. However,
prior methods often struggle with a vocabulary mismatch between pretrained
models, and LM cannot be directly utilized as they diverge from its NLU
formulation. In this study, we propose a three-pass end-to-end (E2E) SLU system
that effectively integrates ASR and LM subnetworks into the SLU formulation for
sequence generation tasks. In the first pass, our architecture predicts ASR
transcripts using the ASR subnetwork. This is followed by the LM subnetwork,
which makes an initial SLU prediction. Finally, in the third pass, the
deliberation subnetwork conditions on representations from the ASR and LM
subnetworks to make the final prediction. Our proposed three-pass SLU system
shows improved performance over cascaded and E2E SLU models on two benchmark
SLU datasets, SLURP and SLUE, especially on acoustically challenging
utterances.

我们提出了一个三步走的端到端 SLU 系统，将预训练的语音识别模型和语言模型有效地集成到 SLU 框架中，以用于序列生成任务。该系统在 ASR 和 LM 子网络的基础上进行预测，最终预测基于 ASR 和 LM 子网络的表示结果。我们的三步走 SLU 系统在两个基准 SLU 数据集（SLURP 和 SLUE）上表现出卓越性能，尤其在具有挑战性的音频对话中的表现更好。

融合预训练的 ASR 和 LM 进行语音理解的序列生成

Integrating Pretrained ASR and LM to Perform Sequence Generation for  Spoken Language Understanding

Memorization presents a challenge for several constrained Natural Language
Generation (NLG) tasks such as Neural Machine Translation (NMT), wherein the
proclivity of neural models to memorize noisy and atypical samples reacts
adversely with the noisy (web crawled) datasets. However, previous studies of
memorization in constrained NLG tasks have only focused on counterfactual
memorization, linking it to the problem of hallucinations. In this work, we
propose a new, inexpensive algorithm for extractive memorization (exact
training data generation under insufficient context) in constrained sequence
generation tasks and use it to study extractive memorization and its effects in
NMT. We demonstrate that extractive memorization poses a serious threat to NMT
reliability by qualitatively and quantitatively characterizing the memorized
samples as well as the model behavior in their vicinity. Based on empirical
observations, we develop a simple algorithm which elicits non-memorized
translations of memorized samples from the same model, for a large fraction of
such samples. Finally, we show that the proposed algorithm could also be
leveraged to mitigate memorization in the model through finetuning. We have
released the code to reproduce our results at
this https URL.

本研究研究了自然语言生成任务中的提取性记忆及其对神经机器翻译的影响，并提出了一种新的算法来生成确切的训练数据以减轻模型中的记忆问题，并通过微调来解决这个问题。

句子生成任务中的记忆提取技术

Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks

Efficient transformer variants with linear time complexity have been
developed to mitigate the quadratic computational overhead of the vanilla
transformer. Among them are low-rank projection methods such as Linformer and
kernel-based Transformers. Despite their unique merits, they usually suffer
from a performance drop comparing with the vanilla transformer on many sequence
generation tasks, and often fail to obtain computation gain when the generation
is short. We propose MemSizer, an approach towards closing the performance gap
while improving the efficiency even with short generation. It projects the
source sequences into lower dimension representations like Linformer, while
enjoying efficient recurrent-style incremental computation similar to
kernel-based transformers. This yields linear computation time and constant
memory complexity at inference time. MemSizer also employs a lightweight
multi-head mechanism which renders the computation as light as a single-head
model. We demonstrate that MemSizer provides an improved balance between
efficiency and accuracy over the vanilla transformer and other efficient
transformer variants in three typical sequence generation tasks, including
machine translation, abstractive text summarization, and language modeling.

提出了 MemSizer 方法来实现低维度表示和类似于基于核的 Transformer 的增量计算，从而在机器翻译、文本摘要和语言建模等三个应用任务中提高了效率和准确性。