The Genomic Foundation Model (GFM) paradigm is expected to facilitate the
extraction of generalizable representations from massive genomic data, thereby
enabling their application across a spectrum of downstream applications.
Despite advancements, a lack of evaluation framework makes it difficult to
ensure equitable assessment due to experimental settings, model intricacy,
benchmark datasets, and reproducibility challenges. In the absence of
standardization, comparative analyses risk becoming biased and unreliable. To
surmount this impasse, we introduce GenBench, a comprehensive benchmarking
suite specifically tailored for evaluating the efficacy of Genomic Foundation
Models. GenBench offers a modular and expandable framework that encapsulates a
variety of state-of-the-art methodologies. Through systematic evaluations of
datasets spanning diverse biological domains with a particular emphasis on both
short-range and long-range genomic tasks, firstly including the three most
important DNA tasks covering Coding Region, Non-Coding Region, Genome
Structure, etc. Moreover, We provide a nuanced analysis of the interplay
between model architecture and dataset characteristics on task-specific
performance. Our findings reveal an interesting observation: independent of the
number of parameters, the discernible difference in preference between the
attention-based and convolution-based models on short- and long-range tasks may
provide insights into the future design of GFM.

通过引入 GenBench，我们为评估 Genomic Foundation Models 的功效提供了一个全面的基准测试套件，该套件特别针对包括编码区域、非编码区域、基因组结构等在内的三个最重要的 DNA 任务。我们的研究结果揭示了一个有趣的观察结果：无论参数数量如何，注意力模型和卷积模型在短程和长程任务上的偏好存在显著差异，这可以为未来的 GFM 设计提供启示。

GenBench: 用于对基因组基础模型进行系统评估的基准套件

GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic  Foundation Models

Advancements in large pre-trained generative models have expanded their
potential as effective data generators in visual recognition. This work delves
into the impact of generative images, primarily comparing paradigms that
harness external data (\ie generative \vs retrieval \vs original).
Our key contributions are: \textbf{1) GenBench Construction:} We devise
\textbf{GenBench}, a broad benchmark comprising 22 datasets with 2548
categories, to appraise generative data across various visual recognition
tasks. \textbf{2) CLER Score:} To address the insufficient correlation of
existing metrics (\eg, FID, CLIP score) with downstream recognition
performance, we propose \textbf{CLER}, a training-free metric indicating
generative data's efficiency for recognition tasks prior to training.
\textbf{3) New Baselines:} Comparisons of generative data with retrieved data
from the same external pool help to elucidate the unique traits of generative
data. \textbf{4) External Knowledge Injection:} By fine-tuning special token
embeddings for each category via Textual Inversion, performance improves across
17 datasets, except when dealing with low-resolution reference images.
Our exhaustive benchmark and analysis spotlight generative data's promise in
visual recognition, while identifying key challenges for future investigation.

本研究探讨了生成图像的影响，通过比较利用外部数据的范例（生成数据 vs 检索数据 vs 原始数据），并提出 GenBench 构建、CLER 评分、新的基准、外部知识注入等关键贡献，突出生成数据在视觉识别中的潜力和未来研究所面临的主要挑战。