In this work, we introduce Libra, a prototype model with a decoupled vision
system on a large language model (LLM). The decoupled vision system decouples
inner-modal modeling and cross-modal interaction, yielding unique visual
information modeling and effective cross-modal comprehension. Libra is trained
through discrete auto-regressive modeling on both vision and language inputs.
Specifically, we incorporate a routed visual expert with a cross-modal bridge
module into a pretrained LLM to route the vision and language flows during
attention computing to enable different attention patterns in inner-modal
modeling and cross-modal interaction scenarios. Experimental results
demonstrate that the dedicated design of Libra achieves a strong MLLM baseline
that rivals existing works in the image-to-text scenario with merely 50 million
training data, providing a new perspective for future multimodal foundation
models. Code is available at this https URL

这项研究介绍了 Libra，这是一个拥有解耦视觉系统的大型语言模型原型。Libra 通过离散的自回归建模，训练视觉和语言输入数据，实现跨模态交互。实验证明，Libra 的专门设计在图像到文本场景中提供了一个强大的 MLLM 基准，仅使用 5000 万个训练数据，为未来的多模态基础模型提供了新的视角。

Libra: 基于大型语言模型构建解耦视觉系统

Libra: Building Decoupled Vision System on Large Language Models

Image captioning models are known to perpetuate and amplify harmful societal
bias in the training set. In this work, we aim to mitigate such gender bias in
image captioning models. While prior work has addressed this problem by forcing
models to focus on people to reduce gender misclassification, it conversely
generates gender-stereotypical words at the expense of predicting the correct
gender. From this observation, we hypothesize that there are two types of
gender bias affecting image captioning models: 1) bias that exploits context to
predict gender, and 2) bias in the probability of generating certain (often
stereotypical) words because of gender. To mitigate both types of gender
biases, we propose a framework, called LIBRA, that learns from synthetically
biased samples to decrease both types of biases, correcting gender
misclassification and changing gender-stereotypical words to more neutral ones.

本文提出名为 LIBRA 的框架，通过学习从合成有偏差的样本中减少性别偏见，从而降低图像字幕模型中的两种性别偏见类型：利用上下文以预测性别的偏见以及因性别导致生成某些（通常是刻板印象的）词语的概率的偏见。