This paper introduces a novel privacy-preservation framework named PFID for
LLMs that addresses critical privacy concerns by localizing user data through
model sharding and singular value decomposition. When users are interacting
with LLM systems, their prompts could be subject to being exposed to
eavesdroppers within or outside LLM system providers who are interested in
collecting users' input. In this work, we proposed a framework to camouflage
user input, so as to alleviate privacy issues. Our framework proposes to place
model shards on the client and the public server, we sent compressed hidden
states instead of prompts to and from servers. Clients have held back
information that can re-privatized the hidden states so that overall system
performance is comparable to traditional LLMs services. Our framework was
designed to be communication efficient, computation can be delegated to the
local client so that the server's computation burden can be lightened. We
conduct extensive experiments on machine translation tasks to verify our
framework's performance.

该论文介绍了一种名为 PFID 的新型隐私保护框架，通过分散模型和奇异值分解的方式，解决了 LLM 系统中关键的隐私问题。

PFID: 面向 LLM 的隐私优先推断委托框架

PFID: Privacy First Inference Delegation Framework for LLMs

Vision Transformers (ViTs) are pivotal for foundational models in scientific
imagery, including Earth science applications, due to their capability to
process large sequence lengths. While transformers for text has inspired
scaling sequence lengths in ViTs, yet adapting these for ViTs introduces unique
challenges. We develop distributed sequence parallelism for ViTs, enabling them
to handle up to 1M tokens. Our approach, leveraging DeepSpeed-Ulysses and
Long-Sequence-Segmentation with model sharding, is the first to apply sequence
parallelism in ViT training, achieving a 94% batch scaling efficiency on 2,048
AMD-MI250X GPUs. Evaluating sequence parallelism in ViTs, particularly in
models up to 10B parameters, highlighted substantial bottlenecks. We countered
these with hybrid sequence, pipeline, tensor parallelism, and flash attention
strategies, to scale beyond single GPU memory limits. Our method significantly
enhances climate modeling accuracy by 20% in temperature predictions, marking
the first training of a transformer model on a full-attention matrix over 188K
sequence length.

利用分布式序列并行化技术以及混合序列、流水线和张量并行化策略，使得 Vision Transformers 在处理大序列长度和训练 10B 参数模型时能够取得更好的性能表现，从而提高气候模型的精度。