This paper presents a new supervised representation learning framework, namely Structured Probabilistic Coding (SPC), to learn compact and informative representations from input related to the target task. SPC is an encoder-only probabilistic coding technology with a structured regularization from the target label space. By extracting compact and informative representations from input related to the target task, SPC can enhance the generalization ability of pre-trained language models for better language understanding. Specifically, the hidden representation is encoded into a Gaussian distribution space, while maximizing the prior entropy of latent representations concerning label space. This technique can simultaneously perform information encoding and task prediction in one module to more fully utilize the effective information from input data, and use variational inference in the output space to reduce randomness and uncertainty. To better control the probability distribution in the latent space, a structured regularization is proposed to promote class-level uniformity in the latent space. With the regularization term, SPC can preserve the Gaussian distribution structure of latent code as well as better cover the hidden space with class uniformly. We conduct evaluations on 12 natural language understanding tasks. The results show that our SPC can effectively improve the performance of pre-trained language models for various classification and regression tasks. Experiments demonstrate that SPC can enhance the generalization capability, robustness to label noise, and clustering quality of output representations.

提出了一种新的监督表示学习框架，即结构化概率编码（SPC），用于从与目标任务相关的输入中学习紧凑且信息丰富的表示。SPC可以提高预先训练语言模型的泛化能力以实现更好的语言理解。通过在高斯分布空间中编码隐藏表示，并同时最大化与标签空间相关的潜在表示的先验熵，SPC可以在一个模块中进行信息编码和任务预测，充分利用输入数据的有效信息并减少随机性和不确定性。结构化正则化促进了概率分布的控制，以保持潜在代码的高斯分布结构，并更好地将隐空间覆盖类别均匀。实验证明，SPC可以提高预先训练语言模型的性能，适用于各种分类和回归任务，并增强了输出表示的泛化能力、标签噪声鲁棒性和聚类质量。