Similar to natural language models, pre-trained genome language models are
proposed to capture the underlying intricacies within genomes with unsupervised
sequence modeling. They have become essential tools for researchers and
practitioners in biology. However, the \textit{hand-crafted} tokenization
policies used in these models may not encode the most discriminative patterns
from the limited vocabulary of genomic data. In this paper, we introduce VQDNA,
a general-purpose framework that renovates genome tokenization from the
perspective of genome vocabulary learning. By leveraging vector-quantized
codebook as \textit{learnable} vocabulary, VQDNA can adaptively tokenize
genomes into \textit{pattern-aware} embeddings in an end-to-end manner. To
further push its limits, we propose Hierarchical Residual Quantization (HRQ),
where varying scales of codebooks are designed in a hierarchy to enrich the
genome vocabulary in a coarse-to-fine manner. Extensive experiments on 32
genome datasets demonstrate VQDNA's superiority and favorable parameter
efficiency compared to existing genome language models. Notably, empirical
analysis of SARS-CoV-2 mutations reveals the fine-grained pattern awareness and
biological significance of learned HRQ vocabulary, highlighting its untapped
potential for broader applications in genomics.

引入了 VQDNA，一个从基因词汇学习的角度改进基因标记化的通用框架；通过利用可学习的词汇的向量量化码书，VQDNA 可以自适应地将基因标记化为具有模式感知的嵌入结果；通过在层次结构中设计不同尺度的码书，提出了层次残差量化方法（HRQ）。在 32 个基因数据集上的广泛实验表明，与现有的基因语言模型相比，VQDNA 具有卓越的性能和有利的参数效率。值得注意的是，对 SARS-CoV-2 突变的经验分析揭示了学习 HRQ 词汇的细粒度模式感知和生物学意义，突显了其在基因组学中更广泛应用的潜力。