The self-attention mechanism utilizes large implicit weight matrices,
programmed through dot product-based activations with very few trainable
parameters, to enable long sequence modeling. In this paper, we investigate the
possibility of discarding residual learning by employing large implicit kernels
to achieve full context interaction at each layer of the network. To accomplish
it, we introduce coordinate-based implicit MLPs as a slow network to generate
hyper-kernels for another fast convolutional network. To get context-varying
weights for fast dynamic encoding, we propose a
$\mathrm{Hyper}\mathcal{Z{\cdot}Z{\cdot}W}$ operator that connects
hyper-kernels ($\mathcal{W}$) and hidden activations ($\mathcal{Z}$) through
simple elementwise multiplication, followed by convolution of $\mathcal{Z}$
using the context-dependent $\mathcal{W}$. Based on this design, we present a
novel Terminator architecture that integrates hyper-kernels of different sizes
to produce multi-branch hidden representations for enhancing the feature
extraction capability of each layer. Additionally, a bottleneck layer is
employed to compress the concatenated channels, allowing only valuable
information to propagate to the subsequent layers. Notably, our model
incorporates several innovative components and exhibits excellent properties,
such as introducing local feedback error for updating the slow network, stable
zero-mean features, faster training convergence, and fewer model parameters.
Extensive experimental results on pixel-level 1D and 2D image classification
benchmarks demonstrate the superior performance of our architecture.

本研究探讨了通过采用大型隐式核函数来实现网络每层的全局上下文交互，并引入了多分支隐藏表示、特征提取等创新组件构建的 Terminator 架构，通过实验证明此架构在像素级 1D 和 2D 图像分类任务中表现出优秀的性能。