In transformer architectures, position encoding primarily provides a sense of
sequence for input tokens. While the original transformer paper's method has
shown satisfactory results in general language processing tasks, there have
been new proposals, such as Rotary Position Embedding (RoPE), for further
improvement. This paper presents geotokens, input components for transformers,
each linked to a specific geological location. Unlike typical language
sequences, for these tokens, the order is not as vital as the geographical
coordinates themselves. To represent the relative position in this context and
to keep a balance between the real world distance and the distance in the
embedding space, we design a position encoding approach drawing from the RoPE
structure but tailored for spherical coordinates.

该论文介绍了一种针对变压器架构的位置编码方法，使用地理位置信息的输入组件，通过球面坐标定制的位置编码方法。与传统的语言序列不同，这些地理位置的相对顺序不如地理坐标本身重要，为了在嵌入空间中保持真实世界距离和距离的平衡，该方法基于 Rotary Position Embedding 结构进行设计。

地理令牌和地理转换器

Geotokens and Geotransformers

Position encoding is the primary mechanism which induces notion of sequential
order for input tokens in transformer architectures. Even though this
formulation in the original transformer paper has yielded plausible performance
for general purpose language understanding and generation, several new
frameworks such as Rotary Position Embedding (RoPE) are proposed for further
enhancement. In this paper, we introduce the notion of "geotokens" which are
input elements for transformer architectures, each representing an information
related to a geological location. Unlike the natural language the sequential
position is not important for the model but the geographical coordinates are.
In order to induce the concept of relative position for such a setting and
maintain the proportion between the physical distance and distance on embedding
space, we formulate a position encoding mechanism based on RoPE architecture
which is adjusted for spherical coordinates.

通过在变压器架构中引入地理标记信息（geotokens），本文基于 RoPE 架构提出了一种适用于球坐标系的位置编码机制，以达到在嵌入空间中保持地理位置与物理距离之间比例关系的目的。