Large Pre-trained Language Models (PLM) have become the most desirable
starting point in the field of NLP, as they have become remarkably good at
solving many individual tasks. Despite such success, in this paper, we argue
that current paradigms of working with PLMs are neglecting a critical aspect of
modeling human intelligence: functional compositionality. Functional
compositionality - the ability to compose learned tasks - has been a
long-standing challenge in the field of AI (and many other fields) as it is
considered one of the hallmarks of human intelligence. An illustrative example
of such is cross-lingual summarization, where a bilingual person
(English-French) could directly summarize an English document into French
sentences without having to translate the English document or summary into
French explicitly. We discuss why this matter is an important open problem that
requires further attention from the field. Then, we show that current PLMs
(e.g., GPT-2 and T5) don't have functional compositionality yet and it is far
from human-level generalizability. Finally, we suggest several research
directions that could push the field towards zero-shot functional
compositionality of language models.

本文探讨了预训练语言模型的功能组合性问题，并指出当前的 PLMs（例如 GPT-2 和 T5）还没有具有功能组合性，远未达到人类级别的泛化能力。同时，对于零样本功能组合的语言模型提出了研究方向。