Convolutional neural networks were the standard for solving many computer
vision tasks until recently, when Transformers of MLP-based architectures have
started to show competitive performance. These architectures typically have a
vast number of weights and need to be trained on massive datasets; hence, they
are not suitable for their use in low-data regimes. In this work, we propose a
simple yet effective framework to improve generalization from small amounts of
data. We augment modern CNNs with fully-connected (FC) layers and show the
massive impact this architectural change has in low-data regimes. We further
present an online joint knowledge-distillation method to utilize the extra FC
layers at train time but avoid them during test time. This allows us to improve
the generalization of a CNN-based model without any increase in the number of
weights at test time. We perform classification experiments for a large range
of network backbones and several standard datasets on supervised learning and
active learning. Our experiments significantly outperform the networks without
fully-connected layers, reaching a relative improvement of up to $16\%$
validation accuracy in the supervised setting without adding any extra
parameters during inference.

本文提出了一种简单而有效的框架，将现代卷积神经网络 (CNNs) 与全连接 (FC) 层相结合，展示了这种架构变化在小数据区间的巨大影响。另外，还提出了一种在线联合知识蒸馏方法，以利用额外的 FC 层来提高 CNN 模型的泛化能力，并在测试时避免了它们。我们在超级学习和主动学习的几个标准数据集上进行了分类实验，实验结果明显优于没有全连接层的网络，在有监督设置下准确率最高可达到 16%。

全连接层在低数据域中的异常有效性

The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes

Convolutional neural networks excel in image recognition tasks, but this
comes at the cost of high computational and memory complexity. To tackle this
problem, [1] developed a tensor factorization framework to compress
fully-connected layers. In this paper, we focus on compressing convolutional
layers. We show that while the direct application of the tensor framework [1]
to the 4-dimensional kernel of convolution does compress the layer, we can do
better. We reshape the convolutional kernel into a tensor of higher order and
factorize it. We combine the proposed approach with the previous work to
compress both convolutional and fully-connected layers of a network and achieve
80x network compression rate with 1.1% accuracy drop on the CIFAR-10 dataset.

本文提出了一种基于张量分解的卷积层参数压缩方法，并将其与之前提出的压缩全连接层的方法结合，成功实现了在 CIFAR-10 数据集上 80 倍的网络压缩和 1.1% 的精度损失。

终极张量化：压缩卷积层和全连接层

Ultimate tensorization: compressing convolutional and FC layers alike

Deep neural networks currently demonstrate state-of-the-art performance in
several domains. At the same time, models of this class are very demanding in
terms of computational resources. In particular, a large amount of memory is
required by commonly used fully-connected layers, making it hard to use the
models on low-end devices and stopping the further increase of the model size.
In this paper we convert the dense weight matrices of the fully-connected
layers to the Tensor Train format such that the number of parameters is reduced
by a huge factor and at the same time the expressive power of the layer is
preserved. In particular, for the Very Deep VGG networks we report the
compression factor of the dense weight matrix of a fully-connected layer up to
200000 times leading to the compression factor of the whole network up to 7
times.

本论文研究了如何使用 Tensor Train 格式转换全连接层的矩阵从而减少深度神经网络的参数数量和计算量，试验结论表明该方法可以将全连接层的参数压缩两百万倍，对整个网络的压缩比可达 7 倍。

张量化神经网络

Tensorizing Neural Networks

In this paper we propose and study a technique to reduce the number of
parameters and computation time in fully-connected layers of neural networks
using Kronecker product, at a mild cost of the prediction quality. The
technique proceeds by replacing Fully-Connected layers with so-called Kronecker
Fully-Connected layers, where the weight matrices of the FC layers are
approximated by linear combinations of multiple Kronecker products of smaller
matrices. In particular, given a model trained on SVHN dataset, we are able to
construct a new KFC model with 73\% reduction in total number of parameters,
while the error only rises mildly. In contrast, using low-rank method can only
achieve 35\% reduction in total number of parameters given similar quality
degradation allowance. If we only compare the KFC layer with its counterpart
fully-connected layer, the reduction in the number of parameters exceeds 99\%.
The amount of computation is also reduced as we replace matrix product of the
large matrices in FC layers with matrix products of a few smaller matrices in
KFC layers. Further experiments on MNIST, SVHN and some Chinese Character
recognition models also demonstrate effectiveness of our technique.

本文介绍了一种使用 Kronecker 积在神经网络的全连接层中减少参数和计算时间的技术，该技术将全连接层替换为 Kronecker 全连接层，能够在需要较小的预测质量折衷的情况下使参数总数减少 73%。