Few ideas have enjoyed as large an impact on deep learning as convolution.
For any problem involving pixels or spatial representations, common intuition
holds that convolutional neural networks may be appropriate. In this paper we
show a striking counterexample to this intuition via the seemingly trivial
coordinate transform problem, which simply requires learning a mapping between
coordinates in (x,y) Cartesian space and one-hot pixel space. Although
convolutional networks would seem appropriate for this task, we show that they
fail spectacularly. We demonstrate and carefully analyze the failure first on a
toy problem, at which point a simple fix becomes obvious. We call this solution
CoordConv, which works by giving convolution access to its own input
coordinates through the use of extra coordinate channels. Without sacrificing
the computational and parametric efficiency of ordinary convolution, CoordConv
allows networks to learn either complete translation invariance or varying
degrees of translation dependence, as required by the end task. CoordConv
solves the coordinate transform problem with perfect generalization and 150
times faster with 10--100 times fewer parameters than convolution. This stark
contrast raises the question: to what extent has this inability of convolution
persisted insidiously inside other tasks, subtly hampering performance from
within? A complete answer to this question will require further investigation,
but we show preliminary evidence that swapping convolution for CoordConv can
improve models on a diverse set of tasks. Using CoordConv in a GAN produced
less mode collapse as the transform between high-level spatial latents and
pixels becomes easier to learn. A Faster R-CNN detection model trained on MNIST
showed 24% better IOU when using CoordConv, and in the RL domain agents playing
Atari games benefit significantly from the use of CoordConv layers.

本篇论文通过坐标转换问题展示了卷积神经网络的局限性，提出了 CoordConv 解决方案，它通过在卷积中引入额外的坐标通道，为神经网络提供了输入坐标的信息，从而可以学习任意程度的平移不变性和变化的空间表示。CoordConv 具有比卷积更高效的计算速度和更少的参数，可以有效提高模型的性能。