Federated learning works by aggregating locally computed gradients from
multiple clients, thus enabling collaborative training without sharing private
client data. However, prior work has shown that the data can actually be
recovered by the server using so-called gradient inversion attacks. While these
attacks perform well when applied on images, they are limited in the text
domain and only permit approximate reconstruction of small batches and short
input sequences. In this work, we propose DAGER, the first algorithm to recover
whole batches of input text exactly. DAGER leverages the low-rank structure of
self-attention layer gradients and the discrete nature of token embeddings to
efficiently check if a given token sequence is part of the client data. We use
this check to exactly recover full batches in the honest-but-curious setting
without any prior on the data for both encoder- and decoder-based architectures
using exhaustive heuristic search and a greedy approach, respectively. We
provide an efficient GPU implementation of DAGER and show experimentally that
it recovers full batches of size up to 128 on large language models (LLMs),
beating prior attacks in speed (20x at same batch size), scalability (10x
larger batches), and reconstruction quality (ROUGE-1/2 > 0.99).

DAGER 是一种用于恢复文本数据批的算法，通过利用 self-attention 层的梯度的低秩结构和标记嵌入的离散性质，以精确的方式恢复整个批次的输入文本数据，用于编码器和解码器架构，并在实验中表现出超过 0.99 的 ROUGE-1/2，速度比之前的攻击快了 20 倍，并且在可扩展性上可以处理 10 倍以上的批大小。

DAGER：大型语言模型的精确梯度反转

DAGER: Exact Gradient Inversion for Large Language Models

Despite Multi-modal Large Language Models (MM-LLMs) have made exciting
strides recently, they are still struggling to efficiently model the
interactions among multi-modal inputs and the generation in non-textual
modalities. In this work, we propose TEAL (Tokenize and Embed ALl)}, an
approach to treat the input from any modality as a token sequence and learn a
joint embedding space for all modalities. Specifically, for the input from any
modality, TEAL first discretizes it into a token sequence with the
off-the-shelf tokenizer and embeds the token sequence into a joint embedding
space with a learnable embedding matrix. MM-LLMs just need to predict the
multi-modal tokens autoregressively as the textual LLMs do. Finally, the
corresponding de-tokenizer is applied to generate the output in each modality
based on the predicted token sequence. With the joint embedding space, TEAL
enables the frozen LLMs to perform both understanding and generation tasks
involving non-textual modalities, such as image and audio. Thus, the textual
LLM can just work as an interface and maintain its high performance in textual
understanding and generation. Experiments show that TEAL achieves substantial
improvements in multi-modal understanding, and implements a simple scheme for
multi-modal generations.

使用 TEAL 方法，将来自任何模态的输入处理为一个记号序列，学习所有模态的联合嵌入空间，并通过联合嵌入空间使得多模态的大型语言模型能够在涉及非文本模态的理解和生成任务中执行，如图像和音频。

TEAL: 多模态大型语言模型的令牌化和嵌入全部

TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models

Pre-trained sequence-to-sequence (seq-to-seq) models have significantly
improved the accuracy of several language generation tasks, including
abstractive summarization. Although the fluency of abstractive summarization
has been greatly improved by fine-tuning these models, it is not clear whether
they can also identify the important parts of the source text to be included in
the summary. In this study, we investigated the effectiveness of combining
saliency models that identify the important parts of the source text with the
pre-trained seq-to-seq models through extensive experiments. We also proposed a
new combination model consisting of a saliency model that extracts a token
sequence from a source text and a seq-to-seq model that takes the sequence as
an additional input text. Experimental results showed that most of the
combination models outperformed a simple fine-tuned seq-to-seq model on both
the CNN/DM and XSum datasets even if the seq-to-seq model is pre-trained on
large-scale corpora. Moreover, for the CNN/DM dataset, the proposed combination
model exceeded the previous best-performed model by 1.33 points on ROUGE-L.

本研究通过实验证明，将能够识别源文本重要部分的显著性模型与预先训练的 seq-to-seq 模型相结合，可以在 CNN/DM 和 XSum 数据集上优于预先训练的 seq-to-seq 模型，并在 ROUGE-L 指标上超过以前最佳表现模型。