Existing object recognition models have been shown to lack robustness in
diverse geographical scenarios due to significant domain shifts in design and
context. Class representations need to be adapted to more accurately reflect an
object concept under these shifts. In the absence of training data from target
geographies, we hypothesize that geography-specific descriptive knowledge of
object categories can be leveraged to enhance robustness. For this purpose, we
explore the feasibility of probing a large-language model for
geography-specific object knowledge, and we investigate integrating knowledge
in zero-shot and learnable soft prompting with the CLIP vision-language model.
In particular, we propose a geography knowledge regularization method to ensure
that soft prompts trained on a source set of geographies generalize to an
unseen target set of geographies. Our gains on DollarStreet when generalizing
from a model trained only on data from Europe are as large as +2.8 on countries
from Africa, and +4.6 on the hardest classes. We further show competitive
performance vs. few-shot target training, and provide insights into how
descriptive knowledge captures geographical differences.

现有的目标识别模型在多样化的地理情况下缺乏鲁棒性，由于设计和上下文中存在显著的领域转变。为了更准确地反映这些转变下的目标概念，需要适应类别表示。在没有来自目标地理区域的训练数据的情况下，我们假设地理特定的目标类别的描述性知识可以利用起来增强鲁棒性。为此，我们探索了在零样本学习和可学习的软提示中探测大规模语言模型的地理特定目标知识的可行性，并提出了一种地理知识正则化方法，以确保在一个源地理集合上训练的软提示可以推广到一个未见过的目标地理集合上。我们在 DollarStreet 数据集上从仅在欧洲数据上训练的模型泛化时，得到了大约 + 2.8（非洲国家）和 + 4.6（最困难的类别）的收益。我们进一步展示了与少样本目标训练相竞争的性能，并深入分析了描述性知识如何捕捉地理差异。