The parallels between protein sequences and natural language in their
sequential structures have inspired the application of large language models
(LLMs) to protein understanding. Despite the success of LLMs in NLP, their
effectiveness in comprehending protein sequences remains an open question,
largely due to the absence of datasets linking protein sequences to descriptive
text. Researchers have then attempted to adapt LLMs for protein understanding
by integrating a protein sequence encoder with a pre-trained LLM. However, this
adaptation raises a fundamental question: "Can LLMs, originally designed for
NLP, effectively comprehend protein sequences as a form of language?" Current
datasets fall short in addressing this question due to the lack of a direct
correlation between protein sequences and corresponding text descriptions,
limiting the ability to train and evaluate LLMs for protein understanding
effectively. To bridge this gap, we introduce ProteinLMDataset, a dataset
specifically designed for further self-supervised pretraining and supervised
fine-tuning (SFT) of LLMs to enhance their capability for protein sequence
comprehension. Specifically, ProteinLMDataset includes 17.46 billion tokens for
pretraining and 893,000 instructions for SFT. Additionally, we present
ProteinLMBench, the first benchmark dataset consisting of 944 manually verified
multiple-choice questions for assessing the protein understanding capabilities
of LLMs. ProteinLMBench incorporates protein-related details and sequences in
multiple languages, establishing a new standard for evaluating LLMs' abilities
in protein comprehension. The large language model InternLM2-7B, pretrained and
fine-tuned on the ProteinLMDataset, outperforms GPT-4 on ProteinLMBench,
achieving the highest accuracy score. The dataset and the benchmark are
available at this https URL

通过整合前期训练模型和预训练的语言模型，本研究提出了 ProteinLMDataset 数据集和 ProteinLMBench 基准数据集，用于提升大型语言模型在蛋白质序列理解方面的能力，并通过实验证明其在蛋白质理解能力方面超越了 GPT-4。

大型语言模型用于蛋白质理解的微调数据集和基准评估

A Fine-tuning Dataset and Benchmark for Large Language Models for  Protein Understanding

Protein language models have shown remarkable success in learning biological
information from protein sequences. However, most existing models are limited
by either autoencoding or autoregressive pre-training objectives, which makes
them struggle to handle protein understanding and generation tasks
concurrently. We propose a unified protein language model, xTrimoPGLM, to
address these two types of tasks simultaneously through an innovative
pre-training framework. Our key technical contribution is an exploration of the
compatibility and the potential for joint optimization of the two types of
objectives, which has led to a strategy for training xTrimoPGLM at an
unprecedented scale of 100 billion parameters and 1 trillion training tokens.
Our extensive experiments reveal that 1) xTrimoPGLM significantly outperforms
other advanced baselines in 18 protein understanding benchmarks across four
categories. The model also facilitates an atomic-resolution view of protein
structures, leading to an advanced 3D structural prediction model that
surpasses existing language model-based tools. 2) xTrimoPGLM not only can
generate de novo protein sequences following the principles of natural ones,
but also can perform programmable generation after supervised fine-tuning (SFT)
on curated sequences. These results highlight the substantial capability and
versatility of xTrimoPGLM in understanding and generating protein sequences,
contributing to the evolving landscape of foundation models in protein science.

通过创新的预训练框架，提出了一个统一的蛋白质语言模型 xTrimoPGLM，能够同时处理蛋白质理解和生成任务，拥有超过 1000 亿个参数和 1 万亿个训练标记的前所未有的规模。在 18 个蛋白质理解基准测试中，xTrimoPGLM 明显优于其他先进基准，能够提供蛋白质结构的原子分辨率视图，并超越现有基于语言模型的工具的 3D 结构预测能力。此外，xTrimoPGLM 不仅可以根据自然原理生成全新的蛋白质序列，还可以在有监督的精调后进行可编程的生成。这些结果凸显了 xTrimoPGLM 在理解和生成蛋白质序列方面的重大能力和多功能性，在蛋白质科学基础模型的发展中具有重要意义。

xTrimoPGLM: 语言解码的统一百亿级预训练变压器

xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering  the Language of Protein

Large Language Models (LLMs) have revolutionized the field of natural
language processing, but they fall short in comprehending biological sequences
such as proteins. To address this challenge, we propose InstructProtein, an
innovative LLM that possesses bidirectional generation capabilities in both
human and protein languages: (i) taking a protein sequence as input to predict
its textual function description and (ii) using natural language to prompt
protein sequence generation. To achieve this, we first pre-train an LLM on both
protein and natural language corpora, enabling it to comprehend individual
languages. Then supervised instruction tuning is employed to facilitate the
alignment of these two distinct languages. Herein, we introduce a knowledge
graph-based instruction generation framework to construct a high-quality
instruction dataset, addressing annotation imbalance and instruction deficits
in existing protein-text corpus. In particular, the instructions inherit the
structural relations between proteins and function annotations in knowledge
graphs, which empowers our model to engage in the causal modeling of protein
functions, akin to the chain-of-thought processes in natural languages.
Extensive experiments on bidirectional protein-text generation tasks show that
InstructProtein outperforms state-of-the-art LLMs by large margins. Moreover,
InstructProtein serves as a pioneering step towards text-based protein function
prediction and sequence design, effectively bridging the gap between protein
and human language understanding.

InstructProtein 是一种具备双向生成能力的语言模型，用于预测蛋白质的功能描述并通过自然语言促使蛋白质序列生成，通过预训练和基于知识图谱的指导数据生成框架，解决了蛋白质序列理解和人类语言理解之间的差距。