This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs)
inference: explicit controllable text generation. Multi-modal LLMs empower
multi-modality understanding with the capability of semantic generation yet
bring less explainability and heavier reliance on prompt contents due to their
autoregressive generative nature. While manipulating prompt formats could
improve outputs, designing specific and precise prompts per task can be
challenging and ineffective. To tackle this issue, we introduce a novel
inference method, Prompt Highlighter, which enables users to highlight specific
prompt spans to interactively control the focus during generation. Motivated by
the classifier-free diffusion guidance, we form regular and unconditional
context pairs based on highlighted tokens, demonstrating that the
autoregressive generation in models can be guided in a classifier-free way.
Notably, we find that, during inference, guiding the models with highlighted
tokens through the attention weights leads to more desired outputs. Our
approach is compatible with current LLMs and VLMs, achieving impressive
customized generation results without training. Experiments confirm its
effectiveness in focusing on input contexts and generating reliable content.
Without tuning on LLaVA-v1.5, our method secured 69.5 in the MMBench test and
1552.5 in MME-perception. The code is available at:
this https URL

该研究介绍了一种新的推理方法，Prompt Highlighter，通过在生成过程中用户可以突出特定的提示段来实现交互式控制，利用突出显示的令牌通过注意力权重指导模型来产生期望的输出。实验结果证实了该方法在关注输入背景和生成可靠内容方面的有效性。