The parallels between protein sequences and natural language in their
sequential structures have inspired the application of large language models
(LLMs) to protein understanding. Despite the success of LLMs in NLP, their
effectiveness in comprehending protein sequences remains an open question,
largely due to the absence of datasets linking protein sequences to descriptive
text. Researchers have then attempted to adapt LLMs for protein understanding
by integrating a protein sequence encoder with a pre-trained LLM. However, this
adaptation raises a fundamental question: "Can LLMs, originally designed for
NLP, effectively comprehend protein sequences as a form of language?" Current
datasets fall short in addressing this question due to the lack of a direct
correlation between protein sequences and corresponding text descriptions,
limiting the ability to train and evaluate LLMs for protein understanding
effectively. To bridge this gap, we introduce ProteinLMDataset, a dataset
specifically designed for further self-supervised pretraining and supervised
fine-tuning (SFT) of LLMs to enhance their capability for protein sequence
comprehension. Specifically, ProteinLMDataset includes 17.46 billion tokens for
pretraining and 893,000 instructions for SFT. Additionally, we present
ProteinLMBench, the first benchmark dataset consisting of 944 manually verified
multiple-choice questions for assessing the protein understanding capabilities
of LLMs. ProteinLMBench incorporates protein-related details and sequences in
multiple languages, establishing a new standard for evaluating LLMs' abilities
in protein comprehension. The large language model InternLM2-7B, pretrained and
fine-tuned on the ProteinLMDataset, outperforms GPT-4 on ProteinLMBench,
achieving the highest accuracy score. The dataset and the benchmark are
available at this https URL

通过整合前期训练模型和预训练的语言模型，本研究提出了 ProteinLMDataset 数据集和 ProteinLMBench 基准数据集，用于提升大型语言模型在蛋白质序列理解方面的能力，并通过实验证明其在蛋白质理解能力方面超越了 GPT-4。