Structure-based protein design has attracted increasing interest, with
numerous methods being introduced in recent years. However, a universally
accepted method for evaluation has not been established, since the wet-lab
validation can be overly time-consuming for the development of new algorithms,
and the $\textit{in silico}$ validation with recovery and perplexity metrics is
efficient but may not precisely reflect true foldability. To address this gap,
we introduce two novel metrics: refoldability-based metric, which leverages
high-accuracy protein structure prediction models as a proxy for wet lab
experiments, and stability-based metric, which assesses whether models can
assign high likelihoods to experimentally stable proteins. We curate datasets
from high-quality CATH protein data, high-throughput $\textit{de novo}$
designed proteins, and mega-scale experimental mutagenesis experiments, and in
doing so, present the $\textbf{PDB-Struct}$ benchmark that evaluates both
recent and previously uncompared protein design methods. Experimental results
indicate that ByProt, ProteinMPNN, and ESM-IF perform exceptionally well on our
benchmark, while ESM-Design and AF-Design fall short on the refoldability
metric. We also show that while some methods exhibit high sequence recovery,
they do not perform as well on our new benchmark. Our proposed benchmark paves
the way for a fair and comprehensive evaluation of protein design methods in
the future. Code is available at this https URL

通过引入两个新的度量标准（基于可重复折叠性和稳定性），我们提出了一种新的评估蛋白质设计方法的综合基准，以解决现有评价方法的不足。实验结果表明，ByProt，ProteinMPNN 和 ESM-IF 在我们的基准测试中表现出色，而 ESM-Design 和 AF-Design 在可重复折叠性指标上表现不佳。我们的基准为未来蛋白质设计方法的公正全面评估铺平了道路。