Unlabeled 3D objects present an opportunity to leverage pretrained vision
language models (VLMs) on a range of annotation tasks -- from describing object
semantics to physical properties. An accurate response must take into account
the full appearance of the object in 3D, various ways of phrasing the
question/prompt, and changes in other factors that affect the response. We
present a method to marginalize over any factors varied across VLM queries,
utilizing the VLM's scores for sampled responses. We first show that this
probabilistic aggregation can outperform a language model (e.g., GPT4) for
summarization, for instance avoiding hallucinations when there are contrasting
details between responses. Secondly, we show that aggregated annotations are
useful for prompt-chaining; they help improve downstream VLM predictions (e.g.,
of object material when the object's type is specified as an auxiliary input in
the prompt). Such auxiliary inputs allow ablating and measuring the
contribution of visual reasoning over language-only reasoning. Using these
evaluations, we show how VLMs can approach, without additional training or
in-context learning, the quality of human-verified type and material
annotations on the large-scale Objaverse dataset.

我们提出了一种方法来边际化通过 VLM 查询变化的任何因素，利用采样响应的 VLM 分数。我们展示了这种概率整合可以在摘要中胜过语言模型（如 GPT4），避免了在响应之间存在对比细节时的幻觉。此外，我们展示了聚合注释对于 Prompt-Chaining 是有用的；它们有助于改进下游 VLM 的预测，例如当在提示中将对象类型指定为辅助输入时，提高了对物体材料的预测质量。利用这些评估，我们展示了 VLM 可以在大规模 Objaverse 数据集上接近人工验证的类型和材料注释的质量，而无需额外的训练或上下文学习。