We present a method to train self-binarizing neural networks, that is,
networks that evolve their weights and activations during training to become
binary. To obtain similar binary networks, existing methods rely on the sign
activation function. This function, however, has no gradients