Large language models (LLMs) have had a huge impact on society due to their
impressive capabilities and vast knowledge of the world. Various applications
and tools have been created that allow users to interact with these models in a
black-box scenario. However, one limitation of this scenario is that users
cannot modify the internal knowledge of the model, and the only way to add or
modify internal knowledge is by explicitly mentioning it to the model during
the current interaction. This learning process is called in-context training,
and it refers to training that is confined to the user's current session or
context. In-context learning has significant applications, but also has
limitations that are seldom studied. In this paper, we present a study that
shows how the model can suffer from interference between information that
continually flows in the context, causing it to forget previously learned
knowledge, which can reduce the model's performance. Along with showing the
problem, we propose an evaluation benchmark based on the bAbI dataset.

大型语言模型对社会产生了巨大影响，但黑盒子情景下，用户只能在当前交互中明确提及来添加或修改内部知识，而不具备修改模型内部知识的能力。本文提出了一个研究，展示了模型在不同上下文中持续流动的信息之间可能出现干扰，从而忘记先前学到的知识，导致模型性能下降，并提出了一个基于 bAbI 数据集的评估基准。

基于聊天式大型语言模型的上下文干扰

In-context Interference in Chat-based Large Language Models

Model inversion attacks (MIAs) are aimed at recovering private data from a
target model's training set, which poses a threat to the privacy of deep
learning models. MIAs primarily focus on the white-box scenario where the
attacker has full access to the structure and parameters of the target model.
However, practical applications are black-box, it is not easy for adversaries
to obtain model-related parameters, and various models only output predicted
labels. Existing black-box MIAs primarily focused on designing the optimization
strategy, and the generative model is only migrated from the GAN used in
white-box MIA. Our research is the pioneering study of feasible attack models
in label-only black-box scenarios, to the best of our knowledge.
In this paper, we develop a novel method of MIA using the conditional
diffusion model to recover the precise sample of the target without any extra
optimization, as long as the target model outputs the label. Two primary
techniques are introduced to execute the attack. Firstly, select an auxiliary
dataset that is relevant to the target model task, and the labels predicted by
the target model are used as conditions to guide the training process.
Secondly, target labels and random standard normally distributed noise are
input into the trained conditional diffusion model, generating target samples
with pre-defined guidance strength. We then filter out the most robust and
representative samples. Furthermore, we propose for the first time to use
Learned Perceptual Image Patch Similarity (LPIPS) as one of the evaluation
metrics for MIA, with systematic quantitative and qualitative evaluation in
terms of attack accuracy, realism, and similarity. Experimental results show
that this method can generate similar and accurate data to the target without
optimization and outperforms generators of previous approaches in the
label-only scenario.

本文提出了在标签黑盒情况下使用条件扩散模型的新型 MIA 方法，可以在不需要额外优化的情况下恢复目标的精确样本，并将 Learned Perceptual Image Patch Similarity 作为评估度量之一进行系统定量和定性评估，实验证明，该方法可以在标签黑盒情况下生成与目标相似且准确的数据，并且优于先前方法的生成器

一次难以防御的攻击：通过条件扩散模型实现仅凭标签进行的模型推断

An Indefensible Attack: Label-Only Model Inversion via Conditional  Diffusion Model

Deep neural networks provide unprecedented performance in all image
classification problems, taking advantage of huge amounts of data available for
training. Recent studies, however, have shown their vulnerability to
adversarial attacks, spawning an intense research effort in this field. With
the aim of building better systems, new countermeasures and stronger attacks
are proposed by the day. On the attacker's side, there is growing interest for
the realistic black-box scenario, in which the user has no access to the neural
network parameters. The problem is to design efficient attacks which mislead
the neural network without compromising image quality. In this work, we propose
to perform the black-box attack along a low-distortion path, so as to improve
both the attack efficiency and the perceptual quality of the adversarial image.
Numerical experiments on real-world systems prove the effectiveness of the
proposed approach, both in benchmark classification tasks and in key
applications in biometrics and forensics.

该研究论文介绍了如何在低失真路径上发起黑盒攻击，以提高攻击效率和敌对图像的感知质量，并在真实世界系统上进行了实验，验证了该方法在基准分类任务和生物识别及取证等关键应用中的有效性。