Ribonucleic acid (RNA) plays a variety of crucial roles in fundamental
biological processes. Recently, RNA has become an interesting drug target,
emphasizing the need to improve our understanding of its structures and
functions. Over the years, sequencing technologies have produced an enormous
amount of unlabeled RNA data, which hides important knowledge and potential.
Motivated by the successes of protein language models, we introduce RiboNucleic
Acid Language Model (RiNALMo) to help unveil the hidden code of RNA. RiNALMo is
the largest RNA language model to date with $650$ million parameters
pre-trained on $36$ million non-coding RNA sequences from several available
databases. RiNALMo is able to extract hidden knowledge and capture the
underlying structure information implicitly embedded within the RNA sequences.
RiNALMo achieves state-of-the-art results on several downstream tasks. Notably,
we show that its generalization capabilities can overcome the inability of
other deep learning methods for secondary structure prediction to generalize on
unseen RNA families. The code has been made publicly available on
this https URL

RiNALMo 是最大的 RNA 语言模型，能够从 RNA 序列中提取隐藏的知识并捕捉内在的结构信息，在多个下游任务中取得了 state-of-the-art 的结果，并且具有通用能力能够克服其他深度学习方法在未见过的 RNA 家族上泛化的能力限制。