Coordinate-MLPs are emerging as an effective tool for modeling
multidimensional continuous signals, overcoming many drawbacks associated with
discrete grid-based approximations. However, coordinate-MLPs with ReLU
activations, in their rudimentary form, demonstrate poor performance in
representing signals with high fidelity, promoting the need for positional
embedding layers. Recently, Sitzmann et al. proposed a sinusoidal activation
function that has the capacity to omit positional embedding from
coordinate-MLPs while still preserving high signal fidelity. Despite its
potential, ReLUs are still dominating the space of coordinate-MLPs; we
speculate that this is due to the hyper-sensitivity of networks -- that employ
such sinusoidal activations -- to the initialization schemes. In this paper, we
attempt to broaden the current understanding of the effect of activations in
coordinate-MLPs, and show that there exists a broader class of activations that
are suitable for encoding signals. We affirm that sinusoidal activations are
only a single example in this class, and propose several non-periodic functions
that empirically demonstrate more robust performance against random
initializations than sinusoids. Finally, we advocate for a shift towards
coordinate-MLPs that employ these non-traditional activation functions due to
their high performance and simplicity.

本文研究了在 Coordinate-MLPs 中 activation function 的影响，提出了一类比正弦函数更加鲁棒的非周期函数，并鼓励在 Coordinate-MLPs 中使用这些非传统的激活函数来提高性能和简化模型。