Pre-trained language models, despite their rapid advancements powered by
scale, still fall short of robust commonsense capabilities. And yet, scale
appears to be the winning recipe; after all, the largest models seem to have
acquired the largest amount of commonsense capabilities. Or is it?
In this paper, we investigate the possibility of a seemingly impossible
match: can smaller language models with dismal commonsense capabilities (i.e.,
GPT-2), ever win over models that are orders of magnitude larger and better
(i.e., GPT-3), if the smaller models are powered with novel commonsense
distillation algorithms? The key intellectual question we ask here is whether
it is possible, if at all, to design a learning algorithm that does not benefit
from scale, yet leads to a competitive level of commonsense acquisition. In
this work, we study the generative models of commonsense knowledge, focusing on
the task of generating generics, statements of commonsense facts about everyday
concepts, e.g., birds can fly.
We introduce a novel commonsense distillation framework, I2D2, that loosely
follows the Symbolic Knowledge Distillation of West et al. but breaks the
dependence on the extreme-scale models as the teacher model by two innovations:
(1) the novel adaptation of NeuroLogic Decoding to enhance the generation
quality of the weak, off-the-shelf language models, and (2) self-imitation
learning to iteratively learn from the model's own enhanced commonsense
acquisition capabilities. Empirical results suggest that scale is not the only
way, as novel algorithms can be a promising alternative. Moreover, our study
leads to a new corpus of generics, Gen-A-Tomic, that is of the largest and
highest quality available to date.

本文研究使用新型的常识提取算法来提升较小的语言模型的常识能力，通过符号知识的传递和自我仿真学习，提出了 I2D2 框架并在生成泛化性句子方面进行实证研究，结果表明规模不是影响模型能力的唯一因素，新的算法是可行的。