We consider scenarios where a very accurate predictive model using restricted
features is available at the time of training of a larger, full-featured,
model. This restricted model may be thought of as "side-information", derived
either from an auxiliary exhaustive dataset or on the same dataset, by forcing
the restriction. How can the restricted model be useful to the full model? We
propose an approach for transferring the knowledge of the restricted model to
the full model, by aligning the full model's context-restricted performance
with that of the restricted model's. We call this methodology Induced Model
Matching (IMM) and first illustrate its general applicability by using logistic
regression as a toy example. We then explore IMM's use in language modeling,
the application that initially inspired it, and where it offers an explicit
foundation in contrast to the implicit use of restricted models in techniques
such as noising. We demonstrate the methodology on both LSTM and transformer
full models, using $N$-grams as restricted models. To further illustrate the
potential of the principle whenever it is much cheaper to collect restricted
rather than full information, we conclude with a simple RL example where POMDP
policies can improve learned MDP policies via IMM.

考虑使用限制特征的非常准确的预测模型在训练更大的全功能模型时的情景，通过将全功能模型的限制上下文性能与限制模型的上下文性能进行对齐，我们提出了一种将限制模型的知识转移到全功能模型的方法，称为引导模型匹配（IMM），并通过逻辑回归和语言建模等应用实例阐述了它的通用适用性和显式基础。我们还使用 n-grams 作为限制模型，在 LSTM 和 transformer 全功能模型上演示了该方法的原理，并通过简单的强化学习示例进一步说明当收集限制信息比收集完整信息廉价得多时，该方法的潜力。