In this paper, we propose \textbf{UniCode}, a novel approach within the
domain of multimodal large language models (MLLMs) that learns a unified
codebook to efficiently tokenize visual, text, and potentially other types of
signals. This innovation addresses a critical limitation in existing MLLMs:
their reliance on a text-only codebook, which restricts MLLM's ability to
generate images and texts in a multimodal context. Towards this end, we propose
a language-driven iterative training paradigm, coupled with an in-context
pre-training task we term ``image decompression'', enabling our model to
interpret compressed visual data and generate high-quality images.The unified
codebook empowers our model to extend visual instruction tuning to
non-linguistic generation tasks. Moreover, UniCode is adaptable to diverse
stacked quantization approaches in order to compress visual signals into a more
compact token representation. Despite using significantly fewer parameters and
less data during training, Unicode demonstrates promising capabilities in
visual reconstruction and generation. It also achieves performances comparable
to leading MLLMs across a spectrum of VQA benchmarks.

提出了一种名为 UniCode 的新方法，通过学习统一的码书，能够在多模态的大型语言模型中有效地对视觉、文本和其他类型的信号进行标记化，通过使用语言驱动的迭代训练方法和图像解压缩预训练任务，我们的模型能够解释压缩的视觉数据并生成高质量图像。UniCode 可适应各种堆叠的量化方法，将视觉信号压缩为更紧凑的标记表示。尽管在训练过程中使用的参数和数据少得多，但 UniCode 在视觉重建和生成方面展示出有希望的能力，并在各种 VQA 基准测试中实现了与领先的 MLLM 相当的性能。

UniCode: 学习用于多模态大型语言模型的统一码书

UniCode: Learning a Unified Codebook for Multimodal Large Language  Models

This research presents a novel framework for the compression and
decompression of medical images utilizing the Latent Diffusion Model (LDM). The
LDM represents advancement over the denoising diffusion probabilistic model
(DDPM) with a potential to yield superior image quality while requiring fewer
computational resources in the image decompression process. A possible
application of LDM and Torchvision for image upscaling has been explored using
medical image data, serving as an alternative to traditional image compression
and decompression algorithms. The experimental outcomes demonstrate that this
approach surpasses a conventional file compression algorithm, and convolutional
neural network (CNN) models trained with decompressed files perform comparably
to those trained with original image files. This approach also significantly
reduces dataset size so that it can be distributed with a smaller size, and
medical images take up much less space in medical devices. The research
implications extend to noise reduction in lossy compression algorithms and
substitute for complex wavelet-based lossless algorithms.

这项研究提出了一种利用潜在扩散模型（LDM）对医学图像进行压缩和解压缩的新框架。实验结果表明，该方法优于传统的文件压缩算法，并且使用解压缩文件训练的卷积神经网络（CNN）模型在性能上与使用原始图像文件训练的模型相当。该方法还显著减小了数据集的大小，使其能够以更小的体积进行分发，并且在医疗设备中占用的空间更小。研究还对于有损压缩算法中的降噪和替代复杂的基于小波的无损算法具有重要意义。