In this report, we present the latest model of the Gemini family, Gemini 1.5
Pro, a highly compute-efficient multimodal mixture-of-experts model capable of
recalling and reasoning over fine-grained information from millions of tokens
of context, including multiple long documents and hours of video and audio.
Gemini 1.5 Pro achieves near-perfect recall on long-context retrieval tasks
across modalities, improves the state-of-the-art in long-document QA,
long-video QA and long-context ASR, and matches or surpasses Gemini 1.0 Ultra's
state-of-the-art performance across a broad set of benchmarks. Studying the
limits of Gemini 1.5 Pro's long-context ability, we find continued improvement
in next-token prediction and near-perfect retrieval (>99%) up to at least 10M
tokens, a generational leap over existing models such as Claude 2.1 (200k) and
GPT-4 Turbo (128k). Finally, we highlight surprising new capabilities of large
language models at the frontier; when given a grammar manual for Kalamang, a
language with fewer than 200 speakers worldwide, the model learns to translate
English to Kalamang at a similar level to a person who learned from the same
content.

Gemini 1.5 Pro 是一种高效的多模态混合专家模型，可以在上亿个上下文令牌中回忆和推理出细粒度信息，包括多个长文档和数小时的视频和音频。该模型在多模态长上下文检索任务、长文档问答、长视频问答和长上下文 ASR 等方面取得了接近完美的召回，性能达到了 Gemini 1.0 Ultra 的最先进水平，并在广泛的基准测试中与之匹敌甚至超过。最后，我们强调了大型语言模型在边界领域的令人惊讶的新功能；当给定卡拉曼语的语法手册时，该模型可以学习将英语翻译成卡拉曼语，达到与从同样内容学习的人类相似的水平。

Gemini 1.5: 在数百万词汇的背景下实现多模态理解的突破

Gemini 1.5: Unlocking multimodal understanding across millions of tokens  of context

Retrieval pipelines-an integral component of many machine learning
systems-perform poorly in domains where documents are long (e.g., 10K tokens or
more) and where identifying the relevant document requires synthesizing
information across the entire text. Developing long-context retrieval encoders
suitable for these domains raises three challenges: (1) how to evaluate
long-context retrieval performance, (2) how to pretrain a base language model
to represent both short contexts (corresponding to queries) and long contexts
(corresponding to documents), and (3) how to fine-tune this model for retrieval
under the batch size limitations imposed by GPU memory constraints. To address
these challenges, we first introduce LoCoV1, a novel 12 task benchmark
constructed to measure long-context retrieval where chunking is not possible or
not effective. We next present the M2-BERT retrieval encoder, an 80M parameter
state-space encoder model built from the Monarch Mixer architecture, capable of
scaling to documents up to 32K tokens long. We describe a pretraining data
mixture which allows this encoder to process both short and long context
sequences, and a finetuning approach that adapts this base model to retrieval
with only single-sample batches. Finally, we validate the M2-BERT retrieval
encoder on LoCoV1, finding that it outperforms competitive baselines by up to
23.3 points, despite containing 5-90x fewer parameters.

为了解决长文本检索中的问题，研究人员引入了 LoCoV1 任务基准和 M2-BERT 检索编码器，通过预训练和微调方法，实现了对长文本的高效检索能力。