With careful manipulation, malicious agents can reverse engineer private
information encoded in pre-trained language models. Security concerns motivate
the development of quantum pre-training. In this work, we propose a highly
Portable Quantum Language Model (PQLM) that can easily transmit information to
downstream tasks on classical machines. The framework consists of a cloud PQLM
built with random Variational Quantum Classifiers (VQC) and local models for
downstream applications. We demonstrate the ad hoc portability of the quantum
model by extracting only the word embeddings and effectively applying them to
downstream tasks on classical machines. Our PQLM exhibits comparable
performance to its classical counterpart on both intrinsic evaluation (loss,
perplexity) and extrinsic evaluation (multilingual sentiment analysis accuracy)
metrics. We also perform ablation studies on the factors affecting PQLM
performance to analyze model stability. Our work establishes a theoretical
foundation for a portable quantum pre-trained language model that could be
trained on private data and made available for public use with privacy
protection guarantees.

本文提出了一种高度可移植的量子语言模型 (PQLM)，它可以在经典计算机上向下游任务轻松传输信息，并演示了在经典计算机上将 PQLM 的词嵌入有效地应用于下游任务。该 PQLM 具有与其经典对应物相当的性能，并为量子预训练语言模型的理论基础奠定了基础。