The demands for higher performance and accuracy in neural networks (NNs)
never end. Existing tensor compilation and Neural Architecture Search (NAS)
techniques orthogonally optimize the two goals but actually share many
similarities in their concrete strategies. We exploit such opportunities by
combining the two into one and make a case for Kernel Architecture Search
(KAS). KAS reviews NAS from a system perspective and zooms into a more
fine-grained level to generate neural kernels with both high performance and
good accuracy. To demonstrate the potential of KAS, we build an end-to-end
framework, Canvas, to find high-quality kernels as convolution replacements.
Canvas samples from a rich set of fine-grained primitives to stochastically and
iteratively construct new kernels and evaluate them according to user-specified
constraints. Canvas supports freely adjustable tensor dimension sizes inside
the kernel and uses two levels of solvers to satisfy structural legality and
fully utilize model budgets. The evaluation shows that by replacing standard
convolutions with generated new kernels in common NNs, Canvas achieves average
1.5x speedups compared to the previous state-of-the-art with acceptable
accuracy loss and search efficiency. Canvas verifies the practicability of KAS
by rediscovering many manually designed kernels in the past and producing new
structures that may inspire future machine learning innovations.

本文介绍了一种结合了 Tensor Compilation 和 Neural Architecture Search 的新方法，称为 Kernel Architecture Search (KAS)，以生成高性能和高准确性的神经内核。作者设计并展示了一个名为 Canvas 的端到端框架，用于发现高质量的卷积替代内核，实现了平均 1.5 倍的加速，并且验证了 KAS 的实用性。