Large language models have made significant strides in natural language processing, paving the way for innovative applications including molecular representation and generation. However, most existing single-modality approaches cannot capture the abundant and complex information in molecular data. Here, we introduce GIT-Mol, a multi-modal large language model that integrates the structure Graph, Image, and Text information, including the Simplified Molecular Input Line Entry System (SMILES) and molecular captions. To facilitate the integration of multi-modal molecular data, we propose GIT-Former, a novel architecture capable of mapping all modalities into a unified latent space. Our study develops an innovative any-to-language molecular translation strategy and achieves a 10%-15% improvement in molecular captioning, a 5%-10% accuracy increase in property prediction, and a 20% boost in molecule generation validity compared to baseline or single-modality models.

通过引入GIT-Mol和GIT-Former，我们开发了一种创新的任意类型到语言的分子翻译策略，并在分子字幕生成方面实现了10%-15%的改进，在属性预测方面提高了5%-10%的准确性，并且在分子生成有效性方面提高了20%。

GIT-Mol: 分子科学的多模态大型语言模型与图形、图像和文本