The rapid development of Multimodal Large Language Models (MLLMs) like GPT-4V
has marked a significant step towards artificial general intelligence. Existing
methods mainly focus on aligning vision encoders with LLMs through supervised
fine-tuning (SFT) to endow LLMs with multimodal abilities, making MLLMs'
inherent ability to react to multiple languages progressively deteriorate as
the training process evolves. We empirically find that the imbalanced SFT
datasets, primarily composed of English-centric image-text pairs, lead to
significantly reduced performance in non-English languages. This is due to the
failure of aligning the vision encoder and LLM with multilingual tokens during
the SFT process. In this paper, we introduce Parrot, a novel method that
utilizes textual guidance to drive visual token alignment at the language
level. Parrot makes the visual tokens condition on diverse language inputs and
uses Mixture-of-Experts (MoE) to promote the alignment of multilingual tokens.
Specifically, to enhance non-English visual tokens alignment, we compute the
cross-attention using the initial visual features and textual embeddings, the
result of which is then fed into the MoE router to select the most relevant
experts. The selected experts subsequently convert the initial visual tokens
into language-specific visual tokens. Moreover, considering the current lack of
benchmarks for evaluating multilingual capabilities within the field, we
collect and make available a Massive Multilingual Multimodal Benchmark which
includes 6 languages, 15 categories, and 12,000 questions, named as MMMB. Our
method not only demonstrates state-of-the-art performance on multilingual
MMBench and MMMB, but also excels across a broad range of multimodal tasks.
Both the source code and the training dataset of Parrot will be made publicly
available.

Parrot 是一种新方法，利用文本指导在语言级别驱动视觉令牌对齐，以增强多语言大型语言模型的多模态能力，并提供了一个大规模多语言多模态基准测试数据集（MMMB）。

鹦鹉：多语言视觉指令调整

Parrot: Multilingual Visual Instruction Tuning

The rise of large language models (LLMs) has enabled LLM-based applications
(a.k.a. AI agents or co-pilots), a new software paradigm that combines the
strength of LLM and conventional software. Diverse LLM applications from
different tenants could design complex workflows using multiple LLM requests to
accomplish one task. However, they have to use the over-simplified
request-level API provided by today's public LLM services, losing essential
application-level information. Public LLM services have to blindly optimize
individual LLM requests, leading to sub-optimal end-to-end performance of LLM
applications.
This paper introduces Parrot, an LLM service system that focuses on the
end-to-end experience of LLM-based applications. Parrot proposes Semantic
Variable, a unified abstraction to expose application-level knowledge to public
LLM services. A Semantic Variable annotates an input/output variable in the
prompt of a request, and creates the data pipeline when connecting multiple LLM
requests, providing a natural way to program LLM applications. Exposing
Semantic Variables to the public LLM service allows it to perform conventional
data flow analysis to uncover the correlation across multiple LLM requests.
This correlation opens a brand-new optimization space for the end-to-end
performance of LLM-based applications. Extensive evaluations demonstrate that
Parrot can achieve up to an order-of-magnitude improvement for popular and
practical use cases of LLM applications.

Parrot 是一种面向基于 LLM 的应用程序的 LLM 服务系统，通过引入 Semantic Variable 实现了端到端性能的提升，能够在 LLM 请求之间进行流程设计，并在多个 LLM 请求之间建立数据管道，从而显著改善 LLM 应用程序的性能。

Parrot: 基于语义变量的 LLM 应用的高效服务

Parrot: Efficient Serving of LLM-based Applications with Semantic  Variable

Impressive progress has been made on chat models based on Large Language
Models (LLMs) recently; however, there is a noticeable lag in multi-turn
conversations between open-source chat models (e.g., Alpaca and Vicuna) and the
leading chat models (e.g., ChatGPT and GPT-4). Through a series of analyses, we
attribute the lag to the lack of enough high-quality multi-turn
instruction-tuning data. The available instruction-tuning data for the
community are either single-turn conversations or multi-turn ones with certain
issues, such as non-human-like instructions, less detailed responses, or rare
topic shifts. In this paper, we address these challenges by introducing Parrot,
a highly scalable solution designed to automatically generate high-quality
instruction-tuning data, which are then used to enhance the effectiveness of
chat models in multi-turn conversations. Specifically, we start by training the
Parrot-Ask model, which is designed to emulate real users in generating
instructions. We then utilize Parrot-Ask to engage in multi-turn conversations
with ChatGPT across a diverse range of topics, resulting in a collection of 40K
high-quality multi-turn dialogues (Parrot-40K). These data are subsequently
employed to train a chat model that we have named Parrot-Chat. We demonstrate
that the dialogues gathered from Parrot-Ask markedly outperform existing
multi-turn instruction-following datasets in critical metrics, including topic
diversity, number of turns, and resemblance to human conversation. With only
40K training examples, Parrot-Chat achieves strong performance against other
13B open-source models across a range of instruction-following benchmarks, and
particularly excels in evaluations of multi-turn capabilities. We make all
codes, datasets, and two versions of the Parrot-Ask model based on LLaMA2-13B
and KuaiYii-13B available at this https URL

通过引入 Parrot，一种具有高可扩展性的解决方案，我们在多轮对话中自动生成高质量的指导语料，并利用这些语料来提高聊天模型在多轮对话中的效果。