How can sLLMs efficiently utilize the supervision of LLMs to improve their
generative quality? This question has been well studied in scenarios where
there is no restriction on the number of LLM supervisions one can use, giving
birth to many decoding algorithms that utilize supervision without further
training. However, it is still unclear what is an effective strategy under the
limited supervision scenario, where we assume that no more than a few tokens
can be generated by LLMs. To this end, we develop an algorithm to effectively
aggregate the sLLM and LLM predictions on initial tokens so that the generated
tokens can more accurately condition the subsequent token generation by sLLM
only. Critically, we find that it is essential to adaptively overtrust or
disregard the LLM prediction based on the confidence of the sLLM. Through our
experiments on a wide range of models and datasets, we demonstrate that our
method provides a consistent improvement over conventional decoding strategies.

发展一种能够有效聚合 sLLM 和 LLM 在初始度上的预测结果以便后续由 sLLM 生成的标记能更准确影响，通过对 sLLM 的置信度自适应性地超信任或忽略 LLM 的预测，我们证明我们的方法在各种模型和数据集上提供了一致的改进。

有限制的教师监督下的解码需要理解何时信任教师

Decoding with Limited Teacher Supervision Requires Understanding When to  Trust the Teacher

Having revolutionized natural language processing (NLP) applications, large
language models (LLMs) are expanding into the realm of multimodal inputs. Owing
to their ability to interpret images, multimodal LLMs (MLLMs) have been
primarily used for vision-language tasks. Currently, MLLMs have not yet been
extended for domain-specific visual tasks, which require a more explicit
understanding of visual information. We developed a method to transform
domain-specific visual and vision-language datasets into a unified question
answering format called Visual Question Answering Instruction (VQA-IN), thereby
extending MLLM to domain-specific tasks. The VQA-IN was applied to train
multiple MLLM architectures using smaller versions of LLMs (sLLMs). The
experimental results indicated that the proposed method achieved a high score
metric on domainspecific visual tasks while also maintaining its performance on
vision-language tasks in a multitask manner.

使用大型语言模型和多模态语言模型，我们开发了一种方法将特定领域的视觉和视觉 - 语言数据集转化为统一的问答格式，从而扩展了多模态语言模型用于特定领域任务，实验结果表明该方法在特定领域的视觉任务和视觉 - 语言任务上达到了高分数指标并保持了多任务的性能。