In recent years, the results of view-based 3D shape recognition methods have
saturated, and models with excellent performance cannot be deployed on
memory-limited devices due to their huge size of parameters. To address this
problem, we introduce a compression method based on knowledge distillation for
this field, which largely reduces the number of parameters while preserving
model performance as much as possible. Specifically, to enhance the
capabilities of smaller models, we design a high-performing large model called
Group Multi-view Vision Transformer (GMViT). In GMViT, the view-level ViT first
establishes relationships between view-level features. Additionally, to capture
deeper features, we employ the grouping module to enhance view-level features
into group-level features. Finally, the group-level ViT aggregates group-level
features into complete, well-formed 3D shape descriptors. Notably, in both
ViTs, we introduce spatial encoding of camera coordinates as innovative
position embeddings. Furthermore, we propose two compressed versions based on
GMViT, namely GMViT-simple and GMViT-mini. To enhance the training
effectiveness of the small models, we introduce a knowledge distillation method
throughout the GMViT process, where the key outputs of each GMViT component
serve as distillation targets. Extensive experiments demonstrate the efficacy
of the proposed method. The large model GMViT achieves excellent 3D
classification and retrieval results on the benchmark datasets ModelNet,
ShapeNetCore55, and MCB. The smaller models, GMViT-simple and GMViT-mini,
reduce the parameter size by 8 and 17.6 times, respectively, and improve shape
recognition speed by 1.5 times on average, while preserving at least 90% of the
classification and retrieval performance.

最近，基于视图的三维形状识别方法的结果已经饱和，而具有出色性能的模型由于参数规模巨大无法部署在内存有限的设备上。为了解决这个问题，我们提出了一种基于知识蒸馏的压缩方法，可以在尽可能保持模型性能的同时大大减少参数数量。

多视角组变换器与空间编码的三维形状分析

Group Multi-View Transformer for 3D Shape Analysis with Spatial Encoding

Recurrent Neural Networks (RNNs) have dominated language modeling because of
their superior performance over traditional N-gram based models. In many
applications, a large Recurrent Neural Network language model (RNNLM) or an
ensemble of several RNNLMs is used. These models have large memory footprints
and require heavy computation. In this paper, we examine the effect of applying
knowledge distillation in reducing the model size for RNNLMs. In addition, we
propose a trust regularization method to improve the knowledge distillation
training for RNNLMs. Using knowledge distillation with trust regularization, we
reduce the parameter size to a third of that of the previously published best
model while maintaining the state-of-the-art perplexity result on Penn Treebank
data. In a speech recognition N-bestrescoring task, we reduce the RNNLM model
size to 18.5% of the baseline system, with no degradation in word error
rate(WER) performance on Wall Street Journal data set.

本文通过应用知识蒸馏和信任正则化方法来减小模型大小，从而降低了循环神经网络（RNN）语言模型的计算成本，并保持了该模型在 Penn Treebank 数据集上的最先进困惑度结果，同时在声音识别任务中没有降低单词错误率（WER）。