In this report, we present the latest model of the Gemini family, Gemini 1.5
Pro, a highly compute-efficient multimodal mixture-of-experts model capable of
recalling and reasoning over fine-grained information from millions of tokens
of context, including multiple long documents and hours of video and audio.
Gemini 1.5 Pro achieves near-perfect recall on long-context retrieval tasks
across modalities, improves the state-of-the-art in long-document QA,
long-video QA and long-context ASR, and matches or surpasses Gemini 1.0 Ultra's
state-of-the-art performance across a broad set of benchmarks. Studying the
limits of Gemini 1.5 Pro's long-context ability, we find continued improvement
in next-token prediction and near-perfect retrieval (>99%) up to at least 10M
tokens, a generational leap over existing models such as Claude 2.1 (200k) and
GPT-4 Turbo (128k). Finally, we highlight surprising new capabilities of large
language models at the frontier; when given a grammar manual for Kalamang, a
language with fewer than 200 speakers worldwide, the model learns to translate
English to Kalamang at a similar level to a person who learned from the same
content.

Gemini 1.5 Pro 是一种高效的多模态混合专家模型，可以在上亿个上下文令牌中回忆和推理出细粒度信息，包括多个长文档和数小时的视频和音频。该模型在多模态长上下文检索任务、长文档问答、长视频问答和长上下文 ASR 等方面取得了接近完美的召回，性能达到了 Gemini 1.0 Ultra 的最先进水平，并在广泛的基准测试中与之匹敌甚至超过。最后，我们强调了大型语言模型在边界领域的令人惊讶的新功能；当给定卡拉曼语的语法手册时，该模型可以学习将英语翻译成卡拉曼语，达到与从同样内容学习的人类相似的水平。

Gemini 1.5: 在数百万词汇的背景下实现多模态理解的突破

Gemini 1.5: Unlocking multimodal understanding across millions of tokens  of context

Annotating long-document question answering (long-document QA) pairs is
time-consuming and expensive. To alleviate the problem, it might be possible to
generate long-document QA pairs via unsupervised question answering (UQA)
methods. However, existing UQA tasks are based on short documents, and can
hardly incorporate long-range information. To tackle the problem, we propose a
new task, named unsupervised long-document question answering (ULQA), aiming to
generate high-quality long-document QA instances in an unsupervised manner.
Besides, we propose AttenWalker, a novel unsupervised method to aggregate and
generate answers with long-range dependency so as to construct long-document QA
pairs. Specifically, AttenWalker is composed of three modules, i.e., span
collector, span linker and answer aggregator. Firstly, the span collector takes
advantage of constituent parsing and reconstruction loss to select informative
candidate spans for constructing answers. Secondly, by going through the
attention graph of a pre-trained long-document model, potentially interrelated
text spans (that might be far apart) could be linked together via an
attention-walking algorithm. Thirdly, in the answer aggregator, linked spans
are aggregated into the final answer via the mask-filling ability of a
pre-trained model. Extensive experiments show that AttenWalker outperforms
previous methods on Qasper and NarrativeQA. In addition, AttenWalker also shows
strong performance in the few-shot learning setting.

提出了一个名为 AttenWalker 的新的无监督长文档问答方法，能够生成高质量的长文档 QA 对，该方法在 Qasper 和 NarrativeQA 上明显优于之前的方法，并在少样本学习中也表现良好。