We present Holistic SparseCNN, a sparse convolutional neural network design that simultaneously optimizes convolution layers (for classification speed) and fully connected layers (for model size), while maintaining the accuracy. We directly apply convolutions to tensors without bandwidth-wasting lowering step, which is critical for sparse convolution that is