Multimodal Large Language Models (MLLMs) excel in generating responses based on visual inputs. However, they often suffer from a bias towards generating responses similar to their pretraining corpus, overshadowing the importance of visual information. We treat this bias as a "preference" for pretraining statistics, which hinders the model's grounding in visual input. To mitigate this issue, we propose Bootstrapped Preference Optimization (BPO), which conducts preference learning with datasets containing negative responses bootstrapped from the model itself. Specifically, we propose the following two strategies: 1) using distorted image inputs to the MLLM for eliciting responses that contain signified pretraining bias; 2) leveraging text-based LLM to explicitly inject erroneous but common elements into the original response. Those undesirable responses are paired with original annotated responses from the datasets to construct the preference dataset, which is subsequently utilized to perform preference learning. Our approach effectively suppresses pretrained LLM bias, enabling enhanced grounding in visual inputs. Extensive experimentation demonstrates significant performance improvements across multiple benchmarks, advancing the state-of-the-art in multimodal conversational systems.

通过利用歪曲图像输入和在原始回应中注入错误但常见元素的方式，我们提出了一种能够有效抑制预训练LLM偏见、提升对视觉输入的基础性的方法——Bootstrap Preference Optimization (BPO)，通过构建偏好数据集并进行偏好学习来解决该偏见问题。广泛的实验表明，在多个基准测试中，我们的方法显著提升了性能，推动了多模态会话系统的发展。

加强多模式大型语言模型的启动式优化偏好