Existing methods to fine-tune LLMs, like Adapter, Prefix-tuning, and LoRA,
which introduce extra modules or additional input sequences to inject new
skills or knowledge, may compromise the innate abilities of LLMs. In this
paper, we propose LLaMA-Excitor, a lightweight method that stimulates the LLMs'
potential to better follow instructions by gradually paying more attention to
worthwhile information. Specifically, the LLaMA-Excitor does not directly
change the intermediate hidden state during the self-attention calculation of
the transformer structure. We designed the Excitor block as a bypass module for
the similarity score computation in LLMs' self-attention to reconstruct keys
and change the importance of values by learnable prompts. LLaMA-Excitor ensures
a self-adaptive allocation of additional attention to input instructions, thus
effectively preserving LLMs' pre-trained knowledge when fine-tuning LLMs on
low-quality instruction-following datasets. Furthermore, we unify the modeling
of multi-modal tuning and language-only tuning, extending LLaMA-Excitor to a
powerful visual instruction follower without the need for complex multi-modal
alignment. Our proposed approach is evaluated in language-only and multi-modal
tuning experimental scenarios. Notably, LLaMA-Excitor is the only method that
maintains basic capabilities while achieving a significant improvement (+6%) on
the MMLU benchmark. In the visual instruction tuning, we achieve a new
state-of-the-art image captioning performance of 157.5 CIDEr on MSCOCO, and a
comparable performance (88.39%) on ScienceQA to cutting-edge models with more
parameters and extensive vision-language pertaining.

LLaMA-Excitor 是一种基于轻量级方法的 LLMs 细调技术，通过逐渐更多地关注值得重视的信息，实现对指令的更好遵循，并保持 LLMs 的预训练知识。此方法在低质量指令遵循数据集的 LLMs 细调中表现出自适应分配额外关注的能力，并能在复杂的多模态对齐需求下将其扩展为强大的视觉指令跟踪器。我们的方法在仅语言和多模态调优实验场景中进行评估，以 MMLU 基准测试为例，LLaMA-Excitor 是唯一保持基本能力且实现显著改进（+6%）的方法。在视觉指令调优方面，我们在 MSCOCO 上实现了 157.5 CIDEr 的最新成果，并在 ScienceQA 上获得了与基于更多参数和广泛视觉语言相关模型相当的性能（88.39%）。

LLaMA-Excitor：通过间接特征交互进行通用指令调优

LLaMA-Excitor: General Instruction Tuning via Indirect Feature  Interaction

Mitigating hallucinations of Large Multi-modal Models(LMMs) is crucial to
enhance their reliability for general-purpose assistants. This paper shows that
such hallucinations of LMMs can be significantly exacerbated by preceding
user-system dialogues. To precisely measure this, we first present an
evaluation benchmark by extending popular multi-modal benchmark datasets with
prepended hallucinatory dialogues generated by our novel Adversarial Question
Generator, which can automatically generate image-related yet adversarial
dialogues by adopting adversarial attacks on LMMs. On our benchmark, the
zero-shot performance of state-of-the-art LMMs dropped significantly for both
the VQA and Captioning tasks. Next, we further reveal this hallucination is
mainly due to the prediction bias toward preceding dialogues rather than visual
content. To reduce this bias, we propose Adversarial Instruction Tuning that
robustly fine-tunes LMMs on augmented multi-modal instruction-following
datasets with hallucinatory dialogues. Extensive experiments show that our
proposed approach successfully reduces dialogue hallucination while maintaining
or even improving performance.

本研究证明了大型多模型（LMMs) 出现幻觉的问题可以通过采用对抗式攻击的新颖敌对性问题生成器生成具有幻觉对话的数据集，并且通过对增强的多模式指令遵循数据集进行鲁棒的微调，成功地减少了对话幻觉。

通过对抗性指导调节来缓解大型多模式模型的对话幻觉

Mitigating Dialogue Hallucination for Large Multi-modal Models via  Adversarial Instruction Tuning

We introduce Instruction Document Visual Question Answering (iDocVQA) dataset
and Large Language Document (LLaDoc) model, for training Language-Vision (LV)
models for document analysis and predictions on document images, respectively.
Usually, deep neural networks for the DocVQA task are trained on datasets
lacking instructions. We show that using instruction-following datasets
improves performance. We compare performance across document-related datasets
using the recent state-of-the-art (SotA) Large Language and Vision Assistant
(LLaVA)1.5 as the base model. We also evaluate the performance of the derived
models for object hallucination using the Polling-based Object Probing
Evaluation (POPE) dataset. The results show that instruction-tuning performance
ranges from 11X to 32X of zero-shot performance and from 0.1% to 4.2% over
non-instruction (traditional task) finetuning. Despite the gains, these still
fall short of human performance (94.36%), implying there's much room for
improvement.

介绍了 Instruction Document Visual Question Answering（iDocVQA）数据集和 Large Language Document（LLaDoc）模型，用于训练文档分析的语言 - 视觉（LV）模型和文档图像上的预测。