Reducing hallucination of Large Language Models (LLMs) is imperative for use
in the sciences where reproducibility is crucial. However, LLMs inherently lack
long-term memory, making it a nontrivial, ad hoc, and inevitably biased task to
fine-tune them on domain-specific literature and data. Here we introduce LLaMP,
a multimodal retrieval-augmented generation (RAG) framework of multiple
data-aware reasoning-and-acting (ReAct) agents that dynamically interact with
computational and experimental data on Materials Project (MP). Without
fine-tuning, LLaMP demonstrates an ability to comprehend and integrate various
modalities of materials science concepts, fetch relevant data stores on the
fly, process higher-order data (such as crystal structures and elastic
tensors), and summarize multi-step procedures for solid-state synthesis. We
show that LLaMP effectively corrects errors in GPT-3.5's intrinsic knowledge,
reducing a 5.21% MAPE on frequently-documented bandgaps and a significant
1103.54% MAPE on formation energies -- errors that GPT-3.5 seems to derive from
mixed data sources. Additionally, LLaMP substantially reduces the hallucinated
volumetric strain in a diamond cubic silicon structure from 66.3% to 0. The
proposed framework offers an intuitive and nearly hallucination-free approach
to exploring materials informatics and establishes a pathway for knowledge
distillation and fine-tuning other language models. We envision the framework
as a valuable component for scientific hypotheses and a foundation for future
autonomous laboratories where multiple LLM agents communicate and cooperate
with robotics to drive material synthesis and chemical reactions without
hard-coded human logic and intervention.

在科学领域，LLMs 的幻觉问题是必须解决的，因为可重现性是至关重要的。本文介绍了 LLaMP，这是一个多模态检索增强生成（RAG）框架，由多个数据感知推理和行动（ReAct）代理组成，动态与 Materials Project（MP）上的计算和实验数据交互。LLaMP 在不进行精细调整的情况下，能够理解和整合各种材料科学概念的多模态信息，实时获取相关数据，处理高阶数据（如晶体结构和弹性张量），并对固态合成的多步骤过程进行总结。同时，LLaMP 能有效纠正 GPT-3.5 在频繁文档中的带隙和形成能量方面的错误，并将钻立方硅结构的虚幻体积应变从 66.3％降低到 0。该框架为材料信息学提供了一种直观且几乎无幻觉的方法，并为知识蒸馏和精细调整其他语言模型奠定了基础。我们设想该框架作为科学假设的有价值组成部分，并为未来的自主实验室奠定基础，多个 LLM 代理与机器人通信和协作，驱动材料合成和化学反应，无需硬编码的人类逻辑和干预。

LLaMP: 用于高保真材料知识检索和蒸馏的大型语言模型

LLaMP: Large Language Model Made Powerful for High-fidelity Materials  Knowledge Retrieval and Distillation

Low-shot image classification, where training images are limited or
inaccessible, has benefited from recent progress on pre-trained vision-language
(VL) models with strong generalizability, e.g. CLIP. Prompt learning methods
built with VL models generate text features from the class names that only have
confined class-specific information. Large Language Models (LLMs), with their
vast encyclopedic knowledge, emerge as the complement. Thus, in this paper, we
discuss the integration of LLMs to enhance pre-trained VL models, specifically
on low-shot classification. However, the domain gap between language and vision
blocks the direct application of LLMs. Thus, we propose LLaMP, Large Language
Models as Prompt learners, that produces adaptive prompts for the CLIP text
encoder, establishing it as the connecting bridge. Experiments show that,
compared with other state-of-the-art prompt learning methods, LLaMP yields
better performance on both zero-shot generalization and few-shot image
classification, over a spectrum of 11 datasets.

通过整合大型语言模型（LLMs）提升预训练视觉 - 语言模型（VL）在低样本图像分类中的能力，提出了大型语言模型作为提示学习者（LLaMP）的方法，并在 11 个数据集上的零样本和小样本图像分类任务中取得了更好的性能。