Language models, primarily transformer-based ones, obtained colossal success
in NLP. To be more precise, studies like BERT in NLU and works such as GPT-3
for NLG are very crucial. DNA sequences are very close to natural language in
terms of structure, so if the DNA-related bioinformatics domain is concerned,
discriminative models, like DNABert, exist. Yet, the generative side of the
coin is mainly unexplored to the best of our knowledge. Consequently, we
focused on developing an autoregressive generative language model like GPT-3
for DNA sequences. Because working with whole DNA sequences is challenging
without substantial computational resources, we decided to carry out our study
on a smaller scale, focusing on nucleotide sequences of human genes, unique
parts in DNA with specific functionalities, instead of the whole DNA. This
decision did not change the problem structure a lot due to the fact that both
DNA and genes can be seen as 1D sequences consisting of four different
nucleotides without losing much information and making too much simplification.
First of all, we systematically examined an almost entirely unexplored problem
and observed that RNNs performed the best while simple techniques like N-grams
were also promising. Another beneficial point was learning how to work with
generative models on languages we do not understand, unlike natural language.
How essential using real-life tasks beyond the classical metrics such as
perplexity is observed. Furthermore, checking whether the data-hungry nature of
these models can be changed through selecting a language with minimal
vocabulary size, four owing to four different types of nucleotides, is
examined. The reason for reviewing this was that choosing such a language might
make the problem easier. However, what we observed in this study was it did not
provide that much of a change in the amount of data needed.

开发类似 GPT-3 的自回归生成语言模型用于 DNA 序列，研究了 RNNs 和 N-grams 在处理人类基因中的核苷酸序列时的表现以及选择具有最小词汇量的语言对数据需求的影响。

人类基因核苷酸序列的生成式语言模型

Generative Language Models on Nucleotide Sequences of Human Genes

Recently long range correlations were detected in nucleotide sequences and in
human writings by several authors. We undertake here a systematic investigation
of two books, Moby Dick by H. Melville and Grimm's tales, with respect to the
existence of long range correlations. The analysis is based on the calculation
of entropy like quantities as the mutual information for pairs of letters and
the entropy, the mean uncertainty, per letter. We further estimate the number
of different subwords of a given length $n$. Filtering out the contributions
due to the effects of the finite length of the texts, we find correlations
ranging to a few hundred letters. Scaling laws for the mutual information
(decay with a power law), for the entropy per letter (decay with the inverse
square root of $n$) and for the word numbers (stretched exponential growth with
$n$ and with a power law of the text length) were found.

通过计算信息熵和互信息，筛选掉文本长度的影响，我们发现了存在于人类文本和核苷酸序列中的长时间相关性，并发现了关于互信息，每个字母的熵以及子单词数量的缩放定律。