Knowledge underpins reasoning. Recent research demonstrates that when
relevant knowledge is provided as additional context to commonsense question
answering (QA), it can substantially enhance the performance even on top of
state-of-the-art. The fundamental challenge is where and how to find such
knowledge that is high quality and on point with respect to the question;
knowledge retrieved from knowledge bases are incomplete and knowledge generated
from language models are inconsistent. We present Rainier, or Reinforced
Knowledge Introspector, that learns to generate contextually relevant knowledge
in response to given questions. Our approach starts by imitating knowledge
generated by GPT-3, then learns to generate its own knowledge via reinforcement
learning where rewards are shaped based on the increased performance on the
resulting question answering. Rainier demonstrates substantial and consistent
performance gains when tested over 9 different commonsense benchmarks:
including 5 datasets that are seen during model training, as well as 4 datasets
that are kept unseen. Our work is the first to report that knowledge generated
by models that are orders of magnitude smaller than GPT-3, even without direct
supervision on the knowledge itself, can exceed the quality of commonsense
knowledge elicited from GPT-3.

本文提出了 Rainier 方法，该方法学习在回答给定问题时生成相关上下文知识，经过增强学习后显示出实质和一致的性能增益，超过了 GPT-3 引出的常识知识的质量，并在 9 个不同的常识基准测试中获得了一致的表现提高。