In the rapidly evolving landscape of deep learning, the quest for models that
balance expressivity with computational efficiency has never been more
critical. This paper introduces Orchid, a novel architecture that reimagines
sequence modeling by incorporating a new data-dependent convolution mechanism.
Orchid is designed to address the inherent limitations of traditional attention
mechanisms, particularly their quadratic complexity, without compromising the
ability to capture long-range dependencies and in-context learning. At the core
of Orchid lies the data-dependent convolution layer, which dynamically adjusts
its kernel conditioned on input data using a dedicated conditioning neural
network. We design two simple conditioning networks that maintain shift
equivariance in the adaptive convolution operation. The dynamic nature of
data-dependent convolution kernel, coupled with gating operations, grants
Orchid high expressivity while maintaining efficiency and quasilinear
scalability for long sequences. We rigorously evaluate Orchid across multiple
domains, including language modeling and image classification, to showcase its
performance and generality. Our experiments demonstrate that Orchid
architecture not only outperforms traditional attention-based architectures
such as BERT and Vision Transformers with smaller model sizes, but also extends
the feasible sequence length beyond the limitations of the dense attention
layers. This achievement represents a significant step towards more efficient
and scalable deep learning models for sequence modeling.

Orchid 是一种新颖的架构，通过整合一种新的数据相关卷积机制，重新定义序列建模，以实现表达能力与计算效率之间的平衡。通过在核上采用专用条件神经网络，Orchid 核心部分是数据相关的卷积层，它根据输入数据动态调整其核。通过门控操作和动态的数据相关卷积核，Orchid 可以在保持效率和准线性可扩展性的同时捕捉长程依赖和上下文学习。我们在多个领域对 Orchid 进行了严格评估，包括语言建模和图像分类，以展示其性能和广泛应用性。我们的实验表明，相比于 BERT 和 Vision Transformers 等传统基于注意力机制的架构，Orchid 架构不仅在模型尺寸较小的情况下表现更好，还能将可行的序列长度超出密集注意力层的限制。这一成就代表了更高效、可扩展的序列建模深度学习模型的重要进步。

Orchid：灵活和数据相关的序列建模卷积

Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling

Bridging geometry and topology, curvature is a powerful and expressive
invariant. While the utility of curvature has been theoretically and
empirically confirmed in the context of manifolds and graphs, its
generalization to the emerging domain of hypergraphs has remained largely
unexplored. On graphs, the Ollivier-Ricci curvature measures differences
between random walks via Wasserstein distances, thus grounding a geometric
concept in ideas from probability theory and optimal transport. We develop
ORCHID, a flexible framework generalizing Ollivier-Ricci curvature to
hypergraphs, and prove that the resulting curvatures have favorable theoretical
properties. Through extensive experiments on synthetic and real-world
hypergraphs from different domains, we demonstrate that ORCHID curvatures are
both scalable and useful to perform a variety of hypergraph tasks in practice.

通过将几何和拓扑学结合起来，曲率是一种功能强大且表达力强的不变量。我们开发了 ORCHID，这是一种灵活的框架，可以将 Ollivier-Ricci 曲率推广到超图，并证明结果曲率具有有利的理论特性，在不同领域的合成和实际超图上进行了广泛实验，证明 ORCHID 曲率不仅可扩展性强，而且有用于执行各种超图任务。