Multimodal protein language models (PLMs) integrate sequence and token-based structural information, serving as a powerful foundation for protein modeling, generation, and design. However, the reliance on tokenizing 3D structures into discrete tokens causes substantial loss of fidelity about fine-grained structural details and correlations. In this paper, we systematically elucidate the design space of multimodal PLMs to overcome their limitations. We identify tokenization loss and inaccurate structure token predictions by the PLMs as major bottlenecks. To address these, our proposed design space covers improved generative modeling, structure-aware architectures and representation learning, and data exploration. Our advancements approach finer-grained supervision, demonstrating that token-based multimodal PLMs can achieve robust structural modeling. The effective design methods dramatically improve the structure generation diversity, and notably, folding abilities of our 650M model by reducing the RMSD from 5.52 to 2.36 on PDB testset, even outperforming 3B baselines and on par with the specialized folding models.

本文研究了多模态蛋白质语言模型在蛋白质建模和设计中的局限性，特别是由于将3D结构切分为离散标记造成的细节和相关性损失。我们提出了一种改进的设计空间，包含更好的生成建模和结构感知架构，显著提升了652M模型的结构生成多样性和折叠能力，RMSD从5.52降低至2.36，超越了3B基线，接近专业折叠模型。

阐明多模态蛋白质语言模型的设计空间