Large Language Models (LLMs) have demonstrated remarkable performance in
various tasks and gained significant attention. LLMs are also used for local
sequence transduction tasks, including grammatical error correction (GEC) and
formality style transfer, where most tokens in a source text are kept
unchanged. However, it is inefficient to generate all target tokens because a
prediction error of a target token may cause a catastrophe in predicting
subsequent tokens and because the computational cost grows quadratically with
the target sequence length. This paper proposes to predict a set of edit
operations for the source text for local sequence transduction tasks.
Representing an edit operation with a span of the source text and changed
tokens, we can reduce the length of the target sequence and thus the
computational cost for inference. We apply instruction tuning for LLMs on the
supervision data of edit operations. Experiments show that the proposed method
achieves comparable performance to the baseline in four tasks, paraphrasing,
formality style transfer, GEC, and text simplification, despite reducing the
length of the target text by as small as 21\%. Furthermore, we report that the
instruction tuning with the proposed method achieved the state-of-the-art
performance in the four tasks.

本文提出一种通过预测编辑操作集合来实现本地序列转导任务的方法，并使用指令调整来对大型语言模型进行监督学习，实验证明该方法在常见任务中具备与基线模型相当的性能，同时可将目标文本长度缩小 21%。

利用大型语言模型预测编辑操作来减少序列长度

Reducing Sequence Length by Predicting Edit Operations with Large  Language Models

Local sequence transduction (LST) tasks are sequence transduction tasks where
there exists massive overlapping between the source and target sequences, such
as Grammatical Error Correction (GEC) and spell or OCR correction. Previous
work generally tackles LST tasks with standard sequence-to-sequence (seq2seq)
models that generate output tokens from left to right and suffer from the issue
of unbalanced outputs. Motivated by the characteristic of LST tasks, in this
paper, we propose a simple but versatile approach named Pseudo-Bidirectional
Decoding (PBD) for LST tasks. PBD copies the corresponding representation of
source tokens to the decoder as pseudo future context to enable the decoder to
attends to its bi-directional context. In addition, the bidirectional decoding
scheme and the characteristic of LST tasks motivate us to share the encoder and
the decoder of seq2seq models. The proposed PBD approach provides right side
context information for the decoder and models the inductive bias of LST tasks,
reducing the number of parameters by half and providing good regularization
effects. Experimental results on several benchmark datasets show that our
approach consistently improves the performance of standard seq2seq models on
LST tasks.

本文研究了 Pseudo-Bidirectional Decoding 在解决 Local Sequence Transduction 聚焦于如语法错误纠正或拼写或 OCR 校正等任务上，能够利用源标记的编码器提供的相关信息来减少参数并提供良好的正则化效果。实验证明该方法比标准 seq2seq 模型在 LST 任务上的性能有所提升。

局部序列转换的伪双向解码

Pseudo-Bidirectional Decoding for Local Sequence Transduction

We present a Parallel Iterative Edit (PIE) model for the problem of local
sequence transduction arising in tasks like Grammatical error correction (GEC).
Recent approaches are based on the popular encoder-decoder (ED) model for
sequence to sequence learning. The ED model auto-regressively captures full
dependency among output tokens but is slow due to sequential decoding. The PIE
model does parallel decoding, giving up the advantage of modelling full
dependency in the output, yet it achieves accuracy competitive with the ED
model for four reasons: 1.~predicting edits instead of tokens, 2.~labeling
sequences instead of generating sequences, 3.~iteratively refining predictions
to capture dependencies, and 4.~factorizing logits over edits and their token
argument to harness pre-trained language models like BERT. Experiments on tasks
spanning GEC, OCR correction and spell correction demonstrate that the PIE
model is an accurate and significantly faster alternative for local sequence
transduction.

本文提出一种并行迭代编辑（PIE）模型来解决本地序列转导问题，该模型在很多任务中都表现出竞争力，并且相比于流行的编码器解码器模型，PIE 模型在保持准确性的同时速度更快，其原因主要有：预测编辑而不是令牌，标记序列而不是生成序列，迭代地优化预测以捕获依赖关系，并且将日志分解为编辑和其令牌参数，以利用预先训练的语言模型，例如 BERT。