Recent zero-shot evaluations have highlighted important limitations in the
abilities of language models (LMs) to perform meaning extraction. However, it
is now well known that LMs can demonstrate radical improvements in the presence
of experimental contexts such as in-context examples and instructions. How well
does this translate to previously studied meaning-sensitive tasks? We present a
case-study on the extent to which experimental contexts can improve LMs'
robustness in performing property inheritance -- predicting semantic properties
of novel concepts, a task that they have been previously shown to fail on. Upon
carefully controlling the nature of the in-context examples and the
instructions, our work reveals that they can indeed lead to non-trivial
property inheritance behavior in LMs. However, this ability is inconsistent:
with a minimal reformulation of the task, some LMs were found to pick up on
shallow, non-semantic heuristics from their inputs, suggesting that the
computational principles of semantic property inference are yet to be mastered
by LMs.

在先前研究中，语言模型在执行属性继承任务方面表现不佳，然而我们的研究表明，在实验环境中加入上下文示例和指令可以显著提高语言模型的鲁棒性，但这一能力并不一致，暗示着语言模型在语义属性推理的计算原则方面仍有待改进。