Binary Neural Network (BNN) represents convolution weights with 1-bit values,
which enhances the efficiency of storage and computation. This paper is
motivated by a previously revealed phenomenon that the binary kernels in
successful BNNs are nearly power-law distributed: their values are mostly
clustered into a small number of codewords. This phenomenon encourages us to
compact typical BNNs and obtain further close performance through learning
non-repetitive kernels within a binary kernel subspace. Specifically, we regard
the binarization process as kernel grouping in terms of a binary codebook, and
our task lies in learning to select a smaller subset of codewords from the full
codebook. We then leverage the Gumbel-Sinkhorn technique to approximate the
codeword selection process, and develop the Permutation Straight-Through
Estimator (PSTE) that is able to not only optimize the selection process
end-to-end but also maintain the non-repetitive occupancy of selected
codewords. Experiments verify that our method reduces both the model size and
bit-wise computational costs, and achieves accuracy improvements compared with
state-of-the-art BNNs under comparable budgets.

本研究提出了一种通过学习二元核子空间内的非重复核来压缩典型的二进制神经网络模型的方法，通过 Gumbel-Sinkhorn 技术近似码字选择过程，在可比较的预算下实现了模型大小和计算成本的降低，并获得了与现有技术相比的精度提升。