The privacy concerns associated with the use of Large Language Models (LLMs)
have grown recently with the development of LLMs such as ChatGPT. Differential
Privacy (DP) techniques are explored in existing work to mitigate their privacy
risks at the cost of generalization degradation. Our paper reveals that the
flatness of DP-trained models' loss landscape plays an essential role in the
trade-off between their privacy and generalization. We further propose a
holistic framework to enforce appropriate weight flatness, which substantially
improves model generalization with competitive privacy preservation. It
innovates from three coarse-to-grained levels, including perturbation-aware
min-max optimization on model weights within a layer, flatness-guided sparse
prefix-tuning on weights across layers, and weight knowledge distillation
between DP \& non-DP weights copies. Comprehensive experiments of both
black-box and white-box scenarios are conducted to demonstrate the
effectiveness of our proposal in enhancing generalization and maintaining DP
characteristics. For instance, on text classification dataset QNLI, DP-Flat
achieves similar performance with non-private full fine-tuning but with DP
guarantee under privacy budget $\epsilon=3$, and even better performance given
higher privacy budgets. Codes are provided in the supplement.

在我们的研究中，我们揭示了 Differential Privacy（DP）技术在处理 Large Language Models（LLMs）的隐私和泛化之间的权衡中，DP 训练模型的损失平面的平坦程度起到了关键作用。我们进一步提出了一个全面的框架来强制执行适当的权重平坦度，从而大幅提高模型的泛化能力并保持竞争性的隐私保护。

通过平坦性实现大型语言模型的隐私保护微调

Privacy-preserving Fine-tuning of Large Language Models through Flatness

Multi-document summarization aims to obtain core information from a
collection of documents written on the same topic. This paper proposes a new
holistic framework for unsupervised multi-document extractive summarization.
Our method incorporates the holistic beam search inference method associated
with the holistic measurements, named Subset Representative Index (SRI). SRI
balances the importance and diversity of a subset of sentences from the source
documents and can be calculated in unsupervised and adaptive manners. To
demonstrate the effectiveness of our method, we conduct extensive experiments
on both small and large-scale multi-document summarization datasets under both
unsupervised and adaptive settings. The proposed method outperforms strong
baselines by a significant margin, as indicated by the resulting ROUGE scores
and diversity measures. Our findings also suggest that diversity is essential
for improving multi-document summary performance.

该研究提出了一种新的整体框架用于无监督的多文档摘要，通过综合度量指标 Subset Representative Index（SRI）平衡了源文件中一部分句子的重要性和多样性，并且经过了大规模实验进行了验证，其结果显示该方法在 ROUGE 得分和多样性度量方面明显优于强基准线，并且多样性对于提高多文档摘要性能至关重要。

无监督全局推理下的多文档自动摘要

Unsupervised Multi-document Summarization with Holistic Inference

The efficient computation of viewpoints under consideration of various system
and process constraints is a common challenge that any robot vision system is
confronted with when trying to execute a vision task. Although fundamental
research has provided solid and sound solutions for tackling this problem, a
holistic framework that poses its formal description, considers the
heterogeneity of robot vision systems, and offers an integrated solution
remains unaddressed. Hence, this publication outlines the generation of
viewpoints as a geometrical problem and introduces a generalized theoretical
framework based on Feature-Based Constrained Spaces ($\mathcal{C}$-spaces) as
the backbone for solving it. A $\mathcal{C}$-space can be understood as the
topological space that a viewpoint constraint spans, where the sensor can be
positioned for acquiring a feature while fulfilling the regarded constraint.
The present study demonstrates that many viewpoint constraints can be
efficiently formulated as $\mathcal{C}$-spaces providing geometric,
deterministic, and closed solutions. The introduced $\mathcal{C}$-spaces are
characterized based on generic domain and viewpoint constraints models to ease
the transferability of the present framework to different applications and
robot vision systems. The effectiveness and efficiency of the concepts
introduced are verified on a simulation-based scenario and validated on a real
robot vision system comprising two different sensors.

本研究提出了一种基于 Feature-Based Constrained Spaces (C-spaces) 的广义理论框架，用于解决从几何问题中生成视角所遇到的系统和过程约束的挑战。