Pretrained language models (PLMs) trained on large-scale unlabeled corpus are
typically fine-tuned on task-specific downstream datasets, which have produced
state-of-the-art results on various NLP tasks. However, the data discrepancy
issue in domain and scale makes fine-tuning fail to efficiently capture
task-specific patterns, especially in the low data regime. To address this
issue, we propose Task-guided Disentangled Tuning (TDT) for PLMs, which
enhances the generalization of representations by disentangling task-relevant
signals from the entangled representations. For a given task, we introduce a
learnable confidence model to detect indicative guidance from context, and
further propose a disentangled regularization to mitigate the over-reliance
problem. Experimental results on GLUE and CLUE benchmarks show that TDT gives
consistently better results than fine-tuning with different PLMs, and extensive
analysis demonstrates the effectiveness and robustness of our method. Code is
available at this https URL.

该研究提出一种名为任务引导分离微调的方法，通过从预训练的语言模型中分离特定任务的信号来增强表示的泛化能力，实验结果表明，该方法在自然语言处理方面表现出更好的效果。