Linguistic Steganography (LS) tasks aim to generate steganographic texts
(stego) based on secret information. Only authorized recipients can perceive
the existence of secret information in the texts and accurately extract it,
thereby preserving privacy. However, the controllability of the stego generated
by existing schemes is poor, and the generated stego is difficult to contain
specific discourse characteristics such as style, genre, and theme. As a
result, the stego are often easily detectable, compromising covert
communication. To address these problems, this paper proposes a novel scheme
named LLsM, a generative LS based on a Large Language Model (LLM). We
fine-tuned the LLM LLaMA2 with a large-scale constructed dataset encompassing
rich discourse characteristics, which enables the fine-tuned LLM to generate
texts with specific discourse in a controllable manner. Then the discourse
characteristics are used as guiding information and inputted into the
fine-tuned LLM in the form of Prompt together with secret information. The
candidate pool, derived from sampling and truncation, undergoes range encoding
to ensure the stego imitate natural text distribution. Experiments demonstrate
that LLsM performs superior to prevalent baselines regarding text quality,
statistical analysis, discourse matching, and anti-steganalysis. In particular,
LLsM's MAUVE surpasses that of some baselines by 70%-80%, and its
anti-steganalysis performance is 30%-40% higher. Notably, we also present the
long stego generated by LLsM, showing its potential superiority in long LS
tasks.

利用大型语言模型（LLM）的 LLsM 方案，通过微调代表语言模型的 LLM LLaMA2，并结合特定信息生成具有特定话语特征的控制性隐写文本，该方案在文本质量、统计分析、话语匹配和反隐写分析等方面优于主流基准，特别是在长篇隐写任务中具备潜在的优势。

LLsM：大型语言模型下的生成型语言隐写

LLsM: Generative Linguistic Steganography with Large Language Model

Label Smoothing (LS) improves model generalization through penalizing models
from generating overconfident output distributions. For each training sample
the LS strategy smooths the one-hot encoded training signal by distributing its
distribution mass over the non-ground truth classes. We extend this technique
by considering example pairs, coined PLS. PLS first creates midpoint samples by
averaging random sample pairs and then learns a smoothing distribution during
training for each of these midpoint samples, resulting in midpoints with high
uncertainty labels for training. We empirically show that PLS significantly
outperforms LS, achieving up to 30% of relative classification error reduction.
We also visualize that PLS produces very low winning softmax scores for both in
and out of distribution samples.

本文介绍了标签平滑（LS）和 example pairs 平滑（PLS）两种方法，其中 PLS 方法较 LS 方法在模型泛化和分类错误率降低等方面表现更优，通过平滑处理来生成高不确定性标签的样本用于训练，实验结果证明了 PLS 方法的有效性。