The Genomic Foundation Model (GFM) paradigm is expected to facilitate the
extraction of generalizable representations from massive genomic data, thereby
enabling their application across a spectrum of downstream applications.
Despite advancements, a lack of evaluation framework makes it difficult to
ensure equitable assessment due to experimental settings, model intricacy,
benchmark datasets, and reproducibility challenges. In the absence of
standardization, comparative analyses risk becoming biased and unreliable. To
surmount this impasse, we introduce GenBench, a comprehensive benchmarking
suite specifically tailored for evaluating the efficacy of Genomic Foundation
Models. GenBench offers a modular and expandable framework that encapsulates a
variety of state-of-the-art methodologies. Through systematic evaluations of
datasets spanning diverse biological domains with a particular emphasis on both
short-range and long-range genomic tasks, firstly including the three most
important DNA tasks covering Coding Region, Non-Coding Region, Genome
Structure, etc. Moreover, We provide a nuanced analysis of the interplay
between model architecture and dataset characteristics on task-specific
performance. Our findings reveal an interesting observation: independent of the
number of parameters, the discernible difference in preference between the
attention-based and convolution-based models on short- and long-range tasks may
provide insights into the future design of GFM.

通过引入 GenBench，我们为评估 Genomic Foundation Models 的功效提供了一个全面的基准测试套件，该套件特别针对包括编码区域、非编码区域、基因组结构等在内的三个最重要的 DNA 任务。我们的研究结果揭示了一个有趣的观察结果：无论参数数量如何，注意力模型和卷积模型在短程和长程任务上的偏好存在显著差异，这可以为未来的 GFM 设计提供启示。