Large-scale language models are trained on a massive amount of natural language data that might encode or reflect our private information. With careful manipulation, malicious agents can reverse engineer the training data even if data sanitation and differential privacy algorithms were involved in the pre-training process. In this work, we propose a decentralized training framework to address privacy concerns in training large-scale language models. The framework consists of a cloud quantum language model built with Variational Quantum Classifiers (VQC) for sentence embedding and a local Long-Short Term Memory (LSTM) model. We use both intrinsic evaluation (loss, perplexity) and extrinsic evaluation (downstream sentiment analysis task) to evaluate the performance of our quantum language model. Our quantum model was comparable to its classical counterpart on all the above metrics. We also perform ablation studies to look into the effect of the size of VQC and the size of training data on the performance of the model. Our approach solves privacy concerns without sacrificing downstream task performance. The intractability of quantum operations on classical hardware ensures the confidentiality of the training data and makes it impossible to be recovered by any adversary.

本文提出了一种高度可移植的量子语言模型(PQLM)，它可以在经典计算机上向下游任务轻松传输信息，并演示了在经典计算机上将 PQLM 的词嵌入有效地应用于下游任务。该 PQLM 具有与其经典对应物相当的性能，并为量子预训练语言模型的理论基础奠定了基础。

PQLM -- 多语言去中心化便携式量子语言模型用于隐私保护