Hyperbolic geometry have shown significant potential in modeling complex
structured data, particularly those with underlying tree-like and hierarchical
structures. Despite the impressive performance of various hyperbolic neural
networks across numerous domains, research on adapting the Transformer to
hyperbolic space remains limited. Previous attempts have mainly focused on
modifying self-attention modules in the Transformer. However, these efforts
have fallen short of developing a complete hyperbolic Transformer. This stems
primarily from: (i) the absence of well-defined modules in hyperbolic space,
including linear transformation layers, LayerNorm layers, activation functions,
dropout operations, etc. (ii) the quadratic time complexity of the existing
hyperbolic self-attention module w.r.t the number of input tokens, which
hinders its scalability. To address these challenges, we propose, Hypformer, a
novel hyperbolic Transformer based on the Lorentz model of hyperbolic geometry.
In Hypformer, we introduce two foundational blocks that define the essential
modules of the Transformer in hyperbolic space. Furthermore, we develop a
linear self-attention mechanism in hyperbolic space, enabling hyperbolic
Transformer to process billion-scale graph data and long-sequence inputs for
the first time. Our experimental results confirm the effectiveness and
efficiency of Hypformer across various datasets, demonstrating its potential as
an effective and scalable solution for large-scale data representation and
large models.

在超几何模型中使用洛伦兹模型，我们提出了 Hypformer，这是一种新型的超几何变压器，它有效地处理了大规模数据表示和大型模型的问题。

Hypformer：在双曲空间完全探索高效的双曲变换器

Hypformer: Exploring Efficient Hyperbolic Transformer Fully in  Hyperbolic Space

Hyperbolic neural networks have shown great potential for modeling complex
data. However, existing hyperbolic networks are not completely hyperbolic, as
they encode features in a hyperbolic space yet formalize most of their
operations in the tangent space (a Euclidean subspace) at the origin of the
hyperbolic space. This hybrid method greatly limits the modeling ability of
networks. In this paper, we propose a fully hyperbolic framework to build
hyperbolic networks based on the Lorentz model by adapting the Lorentz
transformations (including boost and rotation) to formalize essential
operations of neural networks. Moreover, we also prove that linear
transformation in tangent spaces used by existing hyperbolic networks is a
relaxation of the Lorentz rotation and does not include the boost, implicitly
limiting the capabilities of existing hyperbolic networks. The experimental
results on four NLP tasks show that our method has better performance for
building both shallow and deep networks. Our code will be released to
facilitate follow-up research.

本文提出了一种基于洛仑兹模型的完全双曲框架，通过调整洛仑兹变换实现神经网络的基本操作，从而解决了现有双曲网络在欧几里德子空间进行操作的问题并证明了现有双曲网络的能力限制，实验结果展示了该方法在 NLP 任务中表现更优。