Large-scale language models such as DNABert and LOGO aim to learn optimal
gene representations and are trained on the entire Human Reference Genome.
However, standard tokenization schemes involve a simple sliding window of
tokens like k-mers that do not leverage any gene-based semantics and thus may
lead to (trivial) masking of easily predictable sequences and subsequently
inefficient Masked Language Modeling (MLM) training. Therefore, we propose a
novel masking algorithm, GeneMask, for MLM training of gene sequences, where we
randomly identify positions in a gene sequence as mask centers and locally
select the span around the mask center with the highest Normalized Pointwise
Mutual Information (NPMI) to mask. We observe that in the absence of
human-understandable semantics in the genomics domain (in contrast, semantic
units like words and phrases are inherently available in NLP), GeneMask-based
models substantially outperform the SOTA models (DNABert and LOGO) over four
benchmark gene sequence classification datasets in five few-shot settings (10
to 1000-shot). More significantly, the GeneMask-based DNABert model is trained
for less than one-tenth of the number of epochs of the original SOTA model. We
also observe a strong correlation between top-ranked PMI tokens and conserved
DNA sequence motifs, which may indicate the incorporation of latent genomic
information. The codes (including trained models) and datasets are made
publicly available at this https URL

提出了一种新的掩码算法 GeneMask，用于基因序列的掩码语言模型训练，该模型在四个基准基因序列分类数据集的五个少样本设置中明显优于当前最先进模型 DNABert 和 LOGO，同时训练时间不到原始模型的十分之一。