In this work, we try to decipher the internal connection of NLP technology
development in the past decades, searching for essence, which rewards us with a
(potential) new learning paradigm for NLP tasks, dubbed as reStructured
Pre-training (RST). In such a paradigm, the role of data will be re-emphasized,
and model pre-training and fine-tuning of downstream tasks are viewed as a
process of data storing and accessing. Based on that, we operationalize the
simple principle that a good storage mechanism should not only have the ability
to cache a large amount of data but also consider the ease of access. We
achieve this by pre-training models over restructured data that consist of a
variety of valuable information instead of raw data after overcoming several
engineering challenges. Experimentally, RST models not only surpass strong
competitors (e.g., T0) on 52/55 popular datasets from a variety of NLP tasks,
but also achieve superior performance in National College Entrance Examination
- English (Gaokao-English),the most authoritative examination in China.
Specifically, the proposed system Qin achieves 40 points higher than the
average scores made by students and 15 points higher than GPT3 with 1/16
parameters. In particular, Qin gets a high score of 138.5 (the full mark is
150) in the 2018 English exam (national paper III). We have released the Gaokao
Benchmark with an online submission platform.
In addition, we test our model in the 2022 College Entrance Examination
English that happened a few days ago (2022.06.08), and it gets a total score of
134 (v.s. GPT3's 108).

本文提出基于数据存储和访问的 reStructured Pre-training (RST) 新学习范式，并基于此原则实现了 Qin 系统。实验结果表明，该方法在各种 NLP 任务中均优于 T0 等竞争对手，并在中国最具权威的高考英语考试中取得了比学生平均成绩高 40 分和比 GPT3 高 15 分的高分。