Since ChatGPT released its API for public use, the number of applications
built on top of commercial large language models (LLMs) increase exponentially.
One popular usage of such models is leveraging its in-context learning ability
and generating responses given user queries leveraging knowledge obtained by
retrieval augmentation. One problem of deploying commercial retrieval-augmented
LLMs is the cost due to the additionally retrieved context that largely
increases the input token size of the LLMs. To mitigate this, we propose a
token compression scheme that includes two methods: summarization compression
and semantic compression. The first method applies a T5-based model that is
fine-tuned by datasets generated using self-instruct containing samples with
varying lengths and reduce token size by doing summarization. The second method
further compresses the token size by removing words with lower impact on the
semantic. In order to adequately evaluate the effectiveness of the proposed
methods, we propose and utilize a dataset called Food-Recommendation DB (FRDB)
focusing on food recommendation for women around pregnancy period or infants.
Our summarization compression can reduce 65% of the retrieval token size with
further 0.3% improvement on the accuracy; semantic compression provides a more
flexible way to trade-off the token size with performance, for which we can
reduce the token size by 20% with only 1.6% of accuracy drop.

使用自学生成的样本进行 Fine-tune，我们提出了两种方法进行文本压缩：摘要压缩和语义压缩。摘要压缩可以减少 65% 的检索令牌大小并提高 0.3% 的准确性，语义压缩可以将令牌大小减少 20%，但只有 1.6% 的准确性损失。