Image segmentation is one of the most fundamental problems in computer vision
and has drawn a lot of attentions due to its vast applications in image
understanding and autonomous driving. However, designing effective and
efficient segmentation neural architectures is a labor-intensive process that
may require lots of trials by human experts. In this paper, we address the
challenge of integrating multi-head self-attention into high resolution
representation CNNs efficiently, by leveraging architecture search. Manually
replacing convolution layers with multi-head self-attention is non-trivial due
to the costly overhead in memory to maintain high resolution. By contrast, we
develop a multi-target multi-branch supernet method, which not only fully
utilizes the advantages of high-resolution features, but also finds the proper
location for placing multi-head self-attention module. Our search algorithm is
optimized towards multiple objective s (e.g., latency and mIoU) and capable of
finding architectures on Pareto frontier with arbitrary number of branches in a
single search. We further present a series of model via Hybrid
Convolutional-Transformer Architecture Search (HyCTAS) method that searched for
the best hybrid combination of light-weight convolution layers and
memory-efficient self-attention layers between branches from different
resolutions and fuse to high resolution for both efficiency and effectiveness.
Extensive experiments demonstrate that HyCTAS outperforms previous methods on
semantic segmentation task. Code and models are available at
https://github.com/MarvinYu1995/HyCTAS.

通过使用多头自注意力结构，结合架构搜索方法，本研究提出了一种高效且有效的高分辨率图像分割神经网络架构，名为 HyCTAS，通过在多个分辨率的分支之间融合轻量级卷积层和内存高效的自注意力层，提高了语义分割任务的性能。