We focus on prediction problems with structured outputs that are subject to
output validity constraints, e.g. pseudocode-to-code translation where the code
must compile. While labeled input-output pairs are expensive to obtain,
"unlabeled" outputs, i.e. outputs without corresponding inputs, are freely
available (e.g. code on GitHub) and provide information about output validity.
Pre-training captures this structure by training a denoiser to denoise
corrupted versions of unlabeled outputs. We first show that standard
fine-tuning after pre-training destroys some of this structure. We then propose
composed fine-tuning, which trains a predictor composed with the pre-trained
denoiser. Importantly, the denoiser is fixed to preserve output structure. Like
standard fine-tuning, the predictor is also initialized with the pre-trained
denoiser. We prove for two-layer ReLU networks that composed fine-tuning
significantly reduces the complexity of the predictor, thus improving
generalization. Empirically, we show that composed fine-tuning improves over
standard fine-tuning on two pseudocode-to-code translation datasets (3% and 6%
relative). The improvement is magnified on out-of-distribution (OOD) examples
(4% and 25% relative), suggesting that reducing predictor complexity improves
OOD extrapolation.

探讨结构化输出的预测问题，研究在输出有效性限制约束情境下的预测，推荐使用先进行预训练再进行 fine-tuning 的方法，可以通过控制预测器的复杂性以提高其在未知数据上的表现。