Recent research has shown that combining Mamba with Transformer architecture,
which has selective state space and quadratic self-attention mechanism,
outperforms using Mamba or Transformer architecture alone in language modeling
tasks. The quadratic self-attention mechanism effectively alleviates the
shortcomings of selective state space in handling long-term dependencies of any
element in the sequence. We propose a position information injection method
that connects the selective state space model with the quadratic attention, and
integrates these two architectures with hybrid experts with cross-sharing
domains, so that we can enjoy the advantages of both. We design a new
architecture with a more biomimetic idea: Observer-Thinker-Conceiver-Expresser
(OTCE), which can compete with well-known medium-scale open-source language
models on a small scale in language modeling tasks.

将 Mamba 与 Transformer 架构相结合，利用二次自注意机制与有选择的状态空间处理长期依赖关系，并通过位置信息注入方法连接两种架构，设计了一种新的 Observer-Thinker-Conceiver-Expresser (OTCE) 架构，在小规模语言建模任务中能够与知名的中等规模开源语言模型竞争。