We explore the possibility of meta-learning for the language-independent
unsupervised tokenization problem for English, Russian, and Chinese. We
implement the meta-learning approach for automatic determination of
hyper-parameters of the unsupervised tokenization model proposed in earlier
works, relying on various human-independent fitness functions such as
normalised anti-entropy, compression factor and cross-split F 1 score, as well
as additive and multiplicative composite combinations of the three metrics,
testing them against the conventional F1 tokenization score. We find a fairly
good correlation between the latter and the additive combination of the former
three metrics for English and Russian. In case of Chinese, we find a
significant correlation between the F 1 score and the compression factor. Our
results suggest the possibility of robust unsupervised tokenization of
low-resource and dead languages and allow us to think about human languages in
terms of the evolution of efficient symbolic communication codes with different
structural optimisation schemes that have evolved in different human cultures.

该研究探讨了对于英语、俄语和汉语的无监督分词问题可行的元学习方法，通过实现基于不同人类独立健身函数的元学习方法以及三个度量的加性和乘性组合来测试不同语言的无监督分词模型并找到了适合每种语言的最佳设置。

无监督跨语言分词的自调参数

Self-tuning hyper-parameters for unsupervised cross-lingual tokenization

In this paper, we address the problem of reducing the memory footprint of
convolutional network architectures. We introduce a vector quantization method
that aims at preserving the quality of the reconstruction of the network
outputs rather than its weights. The principle of our approach is that it
minimizes the loss reconstruction error for in-domain inputs. Our method only
requires a set of unlabelled data at quantization time and allows for efficient
inference on CPU by using byte-aligned codebooks to store the compressed
weights. We validate our approach by quantizing a high performing ResNet-50
model to a memory size of 5MB (20x compression factor) while preserving a top-1
accuracy of 76.1% on ImageNet object classification and by compressing a Mask
R-CNN with a 26x factor.

本文提出一种矢量量化方法，以减小卷积神经网络架构的存储占用，能以较小的内存占用提供高精度的图像识别。

And the Bit Goes Down: 重新审视神经网络量化

And the Bit Goes Down: Revisiting the Quantization of Neural Networks

Modern Convolutional Neural Networks (CNNs) are complex, encompassing
millions of parameters. Their deployment exerts computational, storage and
energy demands, particularly on embedded platforms. Existing approaches to
prune or sparsify CNNs require retraining to maintain inference accuracy. Such
retraining is not feasible in some contexts. In this paper, we explore the
sparsification of CNNs by proposing three model-independent methods. Our
methods are applied on-the-fly and require no retraining. We show that the
state-of-the-art models' weights can be reduced by up to 73% (compression
factor of 3.7x) without incurring more than 5% loss in Top-5 accuracy.
Additional fine-tuning gains only 8% in sparsity, which indicates that our fast
on-the-fly methods are effective.

本论文提出了三种无需重新训练即可进行卷积神经网络稀疏化的方法，研究表明，这些方法能够使得最先进的模型权重减少高达 73％（压缩因子为 3.7 倍），而最多只会损失 5％的 Top-5 精度，附加的微调只能获得 8％的稀疏度，这表明我们的快速稀疏化方法是有效的。

卷积神经网络快速实时非重训练稀疏化

Fast On-the-fly Retraining-free Sparsification of Convolutional Neural  Networks

Deep neural networks currently demonstrate state-of-the-art performance in
several domains. At the same time, models of this class are very demanding in
terms of computational resources. In particular, a large amount of memory is
required by commonly used fully-connected layers, making it hard to use the
models on low-end devices and stopping the further increase of the model size.
In this paper we convert the dense weight matrices of the fully-connected
layers to the Tensor Train format such that the number of parameters is reduced
by a huge factor and at the same time the expressive power of the layer is
preserved. In particular, for the Very Deep VGG networks we report the
compression factor of the dense weight matrix of a fully-connected layer up to
200000 times leading to the compression factor of the whole network up to 7
times.

本论文研究了如何使用 Tensor Train 格式转换全连接层的矩阵从而减少深度神经网络的参数数量和计算量，试验结论表明该方法可以将全连接层的参数压缩两百万倍，对整个网络的压缩比可达 7 倍。