Large pre-trained language models drastically changed the natural language
processing(NLP) landscape. Nowadays, they represent the go-to framework to
tackle diverse NLP tasks, even with a limited number of annotations. However,
using those models in production, either in the cloud or at the edge, remains a
challenge due to the memory footprint and/or inference costs. As an
alternative, recent work on efficient NLP has shown that small weight-efficient
models can reach competitive performance at a fraction of the costs. Here, we
introduce pNLP-Mixer, an embbedding-free model based on the MLP-Mixer
architecture that achieves high weight-efficiency thanks to a novel
linguistically informed projection layer. We evaluate our model on two
multi-lingual semantic parsing datasets, MTOP and multiATIS. On MTOP our
pNLP-Mixer almost matches the performance of mBERT, which has 38 times more
parameters, and outperforms the state-of-the-art of tiny models (pQRNN) with 3
times fewer parameters. On a long-sequence classification task (Hyperpartisan)
our pNLP-Mixer without pretraining outperforms RoBERTa, which has 100 times
more parameters, demonstrating the potential of this architecture.

介绍了一种低内存、低延迟基于 MLP-Mixer 的 pNLP-Mixer 模型，通过在语言学上指导投影层，实现了高效的语义分析，达到了与 38 倍参数的 mBERT 相当的性能。