Language models trained on large text corpora encode rich distributional
information about real-world environments and action sequences. This
information plays a crucial role in current approaches to language processing
tasks like question answering and instruction generation. We describe how to
leverage language models for *non-linguistic* perception and control tasks. Our
approach casts labeling and decision-making as inference in probabilistic
graphical models in which language models parameterize prior distributions over
labels, decisions and parameters, making it possible to integrate uncertain
observations and incomplete background knowledge in a principled way. Applied
to semantic segmentation, household navigation, and activity recognition tasks,
this approach improves predictions on rare, out-of-distribution, and
structurally novel inputs.

使用自然语言处理模型，我们将标签和决策转化成基于概率图模型的推断，进而实现对语义分割、家庭导航和活动识别任务的精确预测，提高对于罕见、不常见以及结构新颖输入的预测能力。