Knowledge graph embedding research has mainly focused on learning continuous
representations of entities and relations tailored towards the link prediction
problem. Recent results indicate an ever increasing predictive ability of
current approaches on benchmark datasets. However, this effectiveness often
comes with the cost of over-parameterization and increased computationally
complexity. The former induces extensive hyperparameter optimization to
mitigate malicious overfitting. The latter magnifies the importance of winning
the hardware lottery. Here, we investigate a remedy for the first problem. We
propose a technique based on Kronecker decomposition to reduce the number of
parameters in a knowledge graph embedding model, while retaining its
expressiveness. Through Kronecker decomposition, large embedding matrices are
split into smaller embedding matrices during the training process. Hence,
embeddings of knowledge graphs are not plainly retrieved but reconstructed on
the fly. The decomposition ensures that elementwise interactions between three
embedding vectors are extended with interactions within each embedding vector.
This implicitly reduces redundancy in embedding vectors and encourages feature
reuse. To quantify the impact of applying Kronecker decomposition on embedding
matrices, we conduct a series of experiments on benchmark datasets. Our
experiments suggest that applying Kronecker decomposition on embedding matrices
leads to an improved parameter efficiency on all benchmark datasets. Moreover,
empirical evidence suggests that reconstructed embeddings entail robustness
against noise in the input knowledge graph. To foster reproducible research, we
provide an open-source implementation of our approach, including training and
evaluation scripts as well as pre-trained models in our knowledge graph
embedding framework (this https URL).

本文基于 Kronecker 分解技术提出了一种方法来减少知识图嵌入模型中的参数数量，拥有更好的参数效率、特征重用和对输入噪声的鲁棒性，在基准数据集上进行了一系列实验表明其有效性。

知识图谱嵌入的克罗内克分解

Kronecker Decomposition for Knowledge Graph Embeddings

GPT is an auto-regressive Transformer-based pre-trained language model which
has attracted a lot of attention in the natural language processing (NLP)
domain due to its state-of-the-art performance in several downstream tasks. The
success of GPT is mostly attributed to its pre-training on huge amount of data
and its large number of parameters (from ~100M to billions of parameters).
Despite the superior performance of GPT (especially in few-shot or zero-shot
setup), this overparameterized nature of GPT can be very prohibitive for
deploying this model on devices with limited computational power or memory.
This problem can be mitigated using model compression techniques; however,
compressing GPT models has not been investigated much in the literature. In
this work, we use Kronecker decomposition to compress the linear mappings of
the GPT-22 model. Our Kronecker GPT-2 model (KnGPT2) is initialized based on
the Kronecker decomposed version of the GPT-2 model and then is undergone a
very light pre-training on only a small portion of the training data with
intermediate layer knowledge distillation (ILKD). Finally, our KnGPT2 is
fine-tuned on down-stream tasks using ILKD as well. We evaluate our model on
both language modeling and General Language Understanding Evaluation benchmark
tasks and show that with more efficient pre-training and similar number of
parameters, our KnGPT2 outperforms the existing DistilGPT2 model significantly.

本研究使用 Kronecker 分解压缩 GPT-22 模型的线性映射，并使用该技术训练得到一种新型的神经语言模型 KnGPT2，该模型在经过有效预训练后，可在具有相同参数数量的情况下，优于现有的 DistilGPT2 模型，在语言建模和通用语言理解评估基准任务上均取得了显著的成绩。

GPT 压缩的 Kronecker 分解

Kronecker Decomposition for GPT Compression

The development of over-parameterized pre-trained language models has made a
significant contribution toward the success of natural language processing.
While over-parameterization of these models is the key to their generalization
power, it makes them unsuitable for deployment on low-capacity devices. We push
the limits of state-of-the-art Transformer-based pre-trained language model
compression using Kronecker decomposition. We use this decomposition for
compression of the embedding layer, all linear mappings in the multi-head
attention, and the feed-forward network modules in the Transformer layer. We
perform intermediate-layer knowledge distillation using the uncompressed model
as the teacher to improve the performance of the compressed model. We present
our KroneckerBERT, a compressed version of the BERT_BASE model obtained using
this framework. We evaluate the performance of KroneckerBERT on well-known NLP
benchmarks and show that for a high compression factor of 19 (5% of the size of
the BERT_BASE model), our KroneckerBERT outperforms state-of-the-art
compression methods on the GLUE. Our experiments indicate that the proposed
model has promising out-of-distribution robustness and is superior to the
state-of-the-art compression methods on SQuAD.

本文利用 Kronecker 分解技术对预训练语言模型进行压缩，在保证模型高通用性的同时，将模型大小压缩至原大小的 5%，并在多个 NLP 基准测试上实现更好的性能和全新的 out-of-distribution 鲁棒性表现。