Convolution is the most time-consuming operation in deep neural network
operations, so its performance is critical to the overall performance of the
neural network. The commonly used methods for convolution on GPU include the
general matrix multiplication (GEMM)-based convolution and the direct
convolution. GEMM-based convolution relies on the im2col algorithm, which
results in a large memory footprint and reduced performance. Direct convolution
does not have the large memory footprint problem, but the performance is not on
par with GEMM-based approach because of the discontinuous memory access. This
paper proposes a window-order-based convolution paradigm on GPU, called im2win,
which not only reduces memory footprint but also offers continuous memory
accesses, resulting in improved performance. Furthermore, we apply a range of
optimization techniques on the convolution CUDA kernel, including shared
memory, tiling, micro-kernel, double buffer, and prefetching. We compare our
implementation with the direct convolution, and PyTorch's GEMM-based
convolution with cuBLAS and six cuDNN-based convolution implementations, with
twelve state-of-the-art DNN benchmarks. The experimental results show that our
implementation 1) uses less memory footprint by 23.1% and achieves 3.5$\times$
TFLOPS compared with cuBLAS, 2) uses less memory footprint by 32.8% and
achieves up to 1.8$\times$ TFLOPS compared with the best performant
convolutions in cuDNN, and 3) achieves up to 155$\times$ TFLOPS compared with
the direct convolution. We further perform an ablation study on the applied
optimization techniques and find that the micro-kernel has the greatest
positive impact on performance.

本文提出了基于 im2win 的卷积范式，旨在通过持续的内存访问提高性能，并经过了优化技术的改进，与其他基于 cuBLAS 和 cuDNN 的卷积实现相比，内存占用少 23.1% 至 32.8%，性能提高了 3.5 倍至 155 倍。

Im2win：GPU 上的高效卷积操作

Im2win: An Efficient Convolution Paradigm on GPU

Convolution is the most expensive operation among neural network operations,
thus its performance is critical to the overall performance of neural networks.
Commonly used convolution approaches, including general matrix multiplication
(GEMM)-based convolution and direct convolution, rely on im2col for data
transformation or do not use data transformation at all, respectively. However,
the im2col data transformation can lead to at least 2$\times$ memory footprint
compared to not using data transformation at all, thus limiting the size of
neural network models running on memory-limited systems. Meanwhile, not using
data transformation usually performs poorly due to nonconsecutive memory access
although it consumes less memory. To solve those problems, we propose a new
memory-efficient data transformation algorithm, called im2win. This algorithm
refactorizes a row of square or rectangle dot product windows of the input
image and flattens unique elements within these windows into a row in the
output tensor, which enables consecutive memory access and data reuse, and thus
greatly reduces the memory overhead. Furthermore, we propose a high-performance
im2win-based convolution algorithm with various optimizations, including
vectorization, loop reordering, etc. Our experimental results show that our
algorithm reduces the memory overhead by average to 41.6% compared to the
PyTorch's convolution implementation based on im2col, and achieves average to
3.6$\times$ and 5.3$\times$ speedup in performance compared to the im2col-based
convolution and not using data transformation, respectively.

提出了一种名为 im2win 的内存高效数据转换算法，能够实现连续的内存访问和数据重用，从而大大降低内存开销，同时通过各种优化实现高性能二维卷积，平均可以将内存开销降低到 41.6％，相对于 im2col 和不使用数据转换，性能平均提高了 3.6 倍和 5.3 倍。