Relational reasoning refers to the ability to infer and understand the
relations between multiple entities. In humans, this ability underpins many
higher cognitive functions, such as problem solving and decision-making, and
has been reliably linked to fluid intelligence. Despite machine learning models
making impressive advances across various domains, such as natural language
processing and vision, the extent to which such models can perform relational
reasoning tasks remains unclear. Here we study the importance of positional
encoding (PE) for relational reasoning in the Transformer, and find that a
learnable PE outperforms all other commonly-used PEs (e.g., absolute, relative,
rotary, etc.). Moreover, we find that when using a PE with a learnable
parameter, the choice of initialization greatly influences the learned
representations and its downstream generalization performance. Specifically, we
find that a learned PE initialized from a small-norm distribution can 1)
uncover ground-truth position information, 2) generalize in the presence of
noisy inputs, and 3) produce behavioral patterns that are consistent with human
performance. Our results shed light on the importance of learning
high-performing and robust PEs during relational reasoning tasks, which will
prove useful for tasks in which ground truth positions are not provided or not
known.

学习可学习的位置编码对关系推理任务的重要性进行研究，发现可学习的位置编码优于常用的其他位置编码，并且初始化对学到的表示和下游泛化性能有着重要影响。结果表明，在没有提供或无法确定真实位置的任务中，学习高性能和鲁棒性的位置编码对关系推理任务非常重要。