Recent work demonstrated Transformers' ability to efficiently copy strings of
exponential sizes, distinguishing them from other architectures. We present the
Causal Relation Network (CausalRN), an all-MLP sequence modeling architecture
that can match Transformers on the copying task. Extending Relation Networks
(RNs), we implemented key innovations to support autoregressive sequence
modeling while maintaining computational feasibility. We discovered that
exponentially-activated RNs are reducible to linear time complexity, and
pre-activation normalization induces an infinitely growing memory pool, similar
to a KV cache. In ablation study, we found both exponential activation and
pre-activation normalization are indispensable for Transformer-level copying.
Our findings provide new insights into what actually constitutes strong
in-context retrieval.

Causal Relation Network (CausalRN) is an MLP sequence modeling architecture that can match Transformers on the copying task, employing key innovations such as exponentially-activated RNs, pre-activation normalization, and providing insights into strong in-context retrieval.