Large Language Models (LLMs) have demonstrated strong capabilities as
knowledge bases and significant in-context reasoning capabilities. However,
previous work challenges their out-of-context reasoning ability, i.e., the
ability to infer information from their training data, instead of from the
context or prompt. This paper focuses on a significant facet of out-of-context
reasoning: Out-of-Context Knowledge Reasoning (OCKR), which is to combine
multiple knowledge to infer new knowledge. We designed a synthetic dataset with
seven representative OCKR tasks to systematically assess the OCKR capabilities
of LLMs. Using this dataset, we evaluated the LLaMA2-13B-chat model and
discovered that its proficiency in this aspect is limited, regardless of
whether the knowledge is trained in a separate or adjacent training settings.
Moreover, training the model to reason with complete reasoning data did not
result in significant improvement. Training the model to perform explicit
knowledge retrieval helps in only one of the tasks, indicating that the model's
limited OCKR capabilities are due to difficulties in retrieving relevant
knowledge. Furthermore, we treat cross-lingual knowledge transfer as a distinct
form of OCKR, and evaluate this ability. Our results show that the evaluated
model also exhibits limited ability in transferring knowledge across languages.
The dataset used in this study is available at
this https URL

本研究探讨了大型语言模型的脱离上下文推理能力，特别关注脱离上下文知识推理和跨语言知识传递，并发现现有模型在这方面的能力有限。