Multimodal Large Language Models (MLLMs) have demonstrated an excellent
understanding of images and 3D data. However, both modalities have shortcomings
in holistically capturing the appearance and geometry of objects. Meanwhile,
Neural Radiance Fields (NeRFs), which encode information within the weights of
a simple Multi-Layer Perceptron (MLP), have emerged as an increasingly
widespread modality that simultaneously encodes the geometry and photorealistic
appearance of objects. This paper investigates the feasibility and
effectiveness of ingesting NeRF into MLLM. We create LLaNA, the first
general-purpose NeRF-language assistant capable of performing new tasks such as
NeRF captioning and Q\&A. Notably, our method directly processes the weights of
the NeRF's MLP to extract information about the represented objects without the
need to render images or materialize 3D data structures. Moreover, we build a
dataset of NeRFs with text annotations for various NeRF-language tasks with no
human intervention. Based on this dataset, we develop a benchmark to evaluate
the NeRF understanding capability of our method. Results show that processing
NeRF weights performs favourably against extracting 2D or 3D representations
from NeRFs.

本研究调查了将 NeRF 引入 MLLM 的可行性和有效性，并创建了第一个通用的 NeRF 语言助手 LLaNA，可以执行新任务，如 NeRF 字幕和问答。结果显示，处理 NeRF 权重的方式相对于从 NeRF 中提取 2D 或 3D 表示方式表现出优越性。