Service providers of large language model (LLM) applications collect user
instructions in the wild and use them in further aligning LLMs with users'
intentions. These instructions, which potentially contain sensitive
information, are annotated by human workers in the process. This poses a new
privacy risk not addressed by the typical private optimization. To this end, we
propose using synthetic instructions to replace real instructions in data
annotation and model fine-tuning. Formal differential privacy is guaranteed by
generating those synthetic instructions using privately fine-tuned generators.
Crucial in achieving the desired utility is our novel filtering algorithm that
matches the distribution of the synthetic instructions to that of the real
ones. In both supervised fine-tuning and reinforcement learning from human
feedback, our extensive experiments demonstrate the high utility of the final
set of synthetic instructions by showing comparable results to real
instructions. In supervised fine-tuning, models trained with private synthetic
instructions outperform leading open-source models such as Vicuna.

通过使用合成指令替代真实指令进行数据注释和模型微调，通过定义差分隐私生成合成指令，匹配合成指令和真实指令的分布来实现所需效用，结果表明，使用合成指令进行监督微调的模型优于开源模型。

大型语言模型对齐的隐私保护指令

Privacy-Preserving Instructions for Aligning Large Language Models

Interaction and navigation defined by natural language instructions in
dynamic environments pose significant challenges for neural agents. This paper
focuses on addressing two challenges: handling long sequence of subtasks, and
understanding complex human instructions. We propose Episodic Transformer
(E.T.), a multimodal transformer that encodes language inputs and the full
episode history of visual observations and actions. To improve training, we
leverage synthetic instructions as an intermediate representation that
decouples understanding the visual appearance of an environment from the
variations of natural language instructions. We demonstrate that encoding the
history with a transformer is critical to solve compositional tasks, and that
pretraining and joint training with synthetic instructions further improve the
performance. Our approach sets a new state of the art on the challenging ALFRED
benchmark, achieving 38.4% and 8.5% task success rates on seen and unseen test
splits.

本文提出了一种名为 Episodic Transformer (E.T.) 的多模态 Transformer，可解决使用自然语言指令进行交互和导航所面临的挑战。通过采用合成指令来改善训练和决策，使得 E.T. 能够处理历史信息，以实现复合式任务，并在具有挑战性的 ALFRED 基准测试中取得了 38.4% 和 8.5% 的任务成功率。