In the growing domain of scientific machine learning, in-context operator
learning has demonstrated notable potential in learning operators from prompted
data during inference stage without weight updates. However, the current
model's overdependence on sensor data, may inadvertently overlook the
invaluable human insight into the operator. To address this, we present a
transformation of in-context operator learning into a multi-modal paradigm. We
propose the use of "captions" to integrate human knowledge about the operator,
expressed through natural language descriptions and equations. We illustrate
how this method not only broadens the flexibility and generality of
physics-informed learning, but also significantly boosts learning performance
and reduces data needs. Furthermore, we introduce a more efficient neural
network architecture for multi-modal in-context operator learning, referred to
as "ICON-LM", based on a language-model-like architecture. We demonstrate the
viability of "ICON-LM" for scientific machine learning tasks, which creates a
new path for the application of language models.

在科学机器学习的不断发展中，上下文操作符学习在推理阶段从提示数据中学习操作符而无需进行权重更新方面显示出显著潜力。然而，当前模型对传感器数据的过度依赖可能会无意中忽视对操作符的宝贵人类洞察力。为了解决这个问题，我们将上下文操作符学习转化为多模态范式的方法。我们提出使用 “标题” 来集成人类关于操作符的知识，通过自然语言描述和方程式来表达。我们说明了这种方法不仅扩展了物理驱动学习的灵活性和广泛性，而且显著提高了学习性能并减少了数据需求。此外，我们介绍了一种更高效的多模态上下文操作符学习神经网络架构，称为 “ICON-LM”，基于类似语言模型的架构。我们展示了 “ICON-LM” 在科学机器学习任务中的可行性，为语言模型的应用开辟了新的道路。