The rise of large language models (LLMs) is revolutionizing information
retrieval, question answering, summarization, and code generation tasks.
However, in addition to confidently presenting factually inaccurate information
at times (known as "hallucinations"), LLMs are also inherently limited by the
number of input and output tokens that can be processed at once, making them
potentially less effective on tasks that require processing a large set or
continuous stream of information. A common approach to reducing the size of
data is through lossless or lossy compression. Yet, in some cases it may not be
strictly necessary to perfectly recover every detail from the original data, as
long as a requisite level of semantic precision or intent is conveyed.
This paper presents three contributions to research on LLMs. First, we
present the results from experiments exploring the viability of approximate
compression using LLMs, focusing specifically on GPT-3.5 and GPT-4 via ChatGPT
interfaces. Second, we investigate and quantify the capability of LLMs to
compress text and code, as well as to recall and manipulate compressed
representations of prompts. Third, we present two novel metrics -- Exact
Reconstructive Effectiveness (ERE) and Semantic Reconstruction Effectiveness
(SRE) -- that quantify the level of preserved intent between text compressed
and decompressed by the LLMs we studied. Our initial results indicate that
GPT-4 can effectively compress and reconstruct text while preserving the
semantic essence of the original text, providing a path to leverage
$\sim$5$\times$ more tokens than present limits allow.

本研究探讨了大型语言模型在近似压缩和语义压缩方面的应用及其效果评价，并提出了二元评价指标：是否精确重构 (ERE) 和语义重构有效性 (SRE)，结果表明 GPT-4 可能有效地压缩和重构文本，并保留原始文本的语义要素。

大型语言模型的语义压缩

Semantic Compression With Large Language Models

A text written using symbols from a given alphabet can be compressed using
the Huffman code, which minimizes the length of the encoded text. It is
necessary, however, to employ a text-specific codebook, i.e. the
symbol-codeword dictionary, to decode the original text. Thus, the compression
performance should be evaluated by the full code length, i.e. the length of the
encoded text plus the length of the codebook. We studied several alphabets for
compressing texts -- letters, n-grams of letters, syllables, words, and
phrases. If only sufficiently short texts are retained, an alphabet of letters
or two-grams of letters is optimal. For the majority of Project Gutenberg
texts, the best alphabet (the one that minimizes the full code length) is given
by syllables or words, depending on the representation of the codebook. Letter
3 and 4-grams, having on average comparable length to syllables/words, perform
noticeably worse than syllables or words. Word 2-grams also are never the best
alphabet, on the account of having a very large codebook. We also show that the
codebook representation is important -- switching from a naive representation
to a compact one significantly improves the matters for alphabets with large
number of symbols, most notably the words.
Thus, meaning-expressing elements of the language (syllables or words)
provide the best compression alphabet.

本研究探讨了不同的字母表对于文本压缩的影响，推荐使用含有意义的音节或单词作为最佳压缩字母表，并优化编码表示以获得更好的压缩效果。

单文本压缩的最优字母表

Optimal alphabet for single text compression

Neural network-based methods for abstractive summarization produce outputs
that are more fluent than other techniques, but which can be poor at content
selection. This work proposes a simple technique for addressing this issue: use
a data-efficient content selector to over-determine phrases in a source
document that should be part of the summary. We use this selector as a
bottom-up attention step to constrain the model to likely phrases. We show that
this approach improves the ability to compress text, while still generating
fluent summaries. This two-step process is both simpler and higher performing
than other end-to-end content selection models, leading to significant
improvements on ROUGE for both the CNN-DM and NYT corpus. Furthermore, the
content selector can be trained with as little as 1,000 sentences, making it
easy to transfer a trained summarizer to a new domain.

本文提出了一种简单的方法来解决神经网络摘要方法中存在的内容选择问题：使用数据有效的内容选择器来过度确定应该作为摘要的来源文档中的短语，通过作为自底向上的注意步骤来约束模型以生成更简洁而流畅的摘要。该方法比其他端到端内容选择模型更简单且性能更高，在 CNN-DM 和 NYT 语料库中表现出显著的 ROUGE 值提高。此外，内容选择器只需要训练 1000 个句子就可以使摘要器轻松地进行领域转移。