We introduce a new way of learning to encode position information for
non-recurrent models, such as Transformer models. Unlike RNN and LSTM, which
contain inductive bias by loading the input tokens sequentially, non-recurrent
models are less sensitive to position. The main reason is that position
information among input units is not inherently encoded, i.e., the models are
permutation equivalent; this problem justifies why all of the existing models
are accompanied by a sinusoidal encoding/embedding layer at the input. However,
this solution has clear limitations: the sinusoidal encoding is not flexible
enough as it is manually designed and does not contain any learnable
parameters, whereas the position embedding restricts the maximum length of
input sequences. It is thus desirable to design a new position layer that
contains learnable parameters to adjust to different datasets and different
architectures. At the same time, we would also like the encodings to
extrapolate in accordance with the variable length of inputs. In our proposed
solution, we borrow from the recent Neural ODE approach, which may be viewed as
a versatile continuous version of a ResNet. This model is capable of modeling
many kinds of dynamical systems. We model the evolution of encoded results
along position index by such a dynamical system, thereby overcoming the above
limitations of existing methods. We evaluate our new position layers on a
variety of neural machine translation and language understanding tasks, the
experimental results show consistent improvements over the baselines.

提出了一种新的位置信息编码方法，使用神经常微分方法对非循环模型（如 Transformer）进行编码，并证明在翻译和理解任务中，该编码方法与已有编码方法相比具有更好的性能。