As large language models (LLMs) become ubiquitous in our daily tasks and
digital interactions, associated privacy risks are increasingly in focus. While
LLM privacy research has primarily focused on the leakage of model training
data, it has recently been shown that the increase in models' capabilities has
enabled LLMs to make accurate privacy-infringing inferences from previously
unseen texts. With the rise of multimodal vision-language models (VLMs),
capable of understanding both images and text, a pertinent question is whether
such results transfer to the previously unexplored domain of benign images
posted online. To investigate the risks associated with the image reasoning
capabilities of newly emerging VLMs, we compile an image dataset with
human-annotated labels of the image owner's personal attributes. In order to
understand the additional privacy risk posed by VLMs beyond traditional human
attribute recognition, our dataset consists of images where the inferable
private attributes do not stem from direct depictions of humans. On this
dataset, we evaluate the inferential capabilities of 7 state-of-the-art VLMs,
finding that they can infer various personal attributes at up to 77.6%
accuracy. Concerningly, we observe that accuracy scales with the general
capabilities of the models, implying that future models can be misused as
stronger adversaries, establishing an imperative for the development of
adequate defenses.

通过研究多模式视觉 - 语言模型（VLMs）的影响力，我们发现这些模型能够从在线发布的图像中推理出个人属性，其中 7 个最先进模型的准确性高达 77.6％，这意味着未来的模型可能被用作更强大的对手，需要开发适当的防御机制。

使用视觉 - 语言模型从图像中推断私密属性

Private Attribute Inference from Images with Vision-Language Models

Large Language Models (LLMs) have demonstrated impressive inferential
capabilities, with numerous research endeavors devoted to enhancing this
capacity through prompting. Despite these efforts, a unified epistemological
foundation is still conspicuously absent. Drawing inspiration from Kant's a
priori philosophy, we propose the UPAR prompting framework, designed to emulate
the structure of human cognition within LLMs. The UPAR framework is delineated
into four phases: "Understand", "Plan", "Act", and "Reflect", enabling the
extraction of structured information from complex contexts, prior planning of
solutions, execution according to plan, and self-reflection. This structure
significantly augments the explainability and accuracy of LLM inference,
producing a human-understandable and inspectable inferential trajectory.
Furthermore, our work offers an epistemological foundation for existing
prompting techniques, allowing for a possible systematic integration of these
methods. With GPT-4, our approach elevates the accuracy from COT baseline of
22.92% to 58.33% in a challenging subset of GSM8K, and from 67.91% to 75.40% in
the causal judgment task.

借鉴康德的先验哲学，我们提出了 UPAR 提示框架，通过四个阶段（理解、计划、执行、反思）模拟人类认知结构，在提取结构化信息、先前规划解决方案、按计划执行和自我反思方面，显著增强了 LLM 推理的可解释性和准确性。此外，我们的工作为现有提示技术建立了认识论基础，为这些方法的系统整合提供可能。在 GSM8K 的具有挑战性的子集中，与 COT 基线的 22.92% 相比，我们的方法使准确率提升至 58.33%；在因果判断任务中，准确率从 67.91% 提高到 75.40%。