Multimodal Large Language Models (mLLMs) are trained on a large amount of
text-image data. While most mLLMs are trained on caption-like data only,
Alayrac et al. [2022] showed that additionally training them on interleaved
sequences of text and images can lead to the emergence of in-context learning
capabilities. However, the dataset they used, M3W, is not public and is only in
English. There have been attempts to reproduce their results but the released
datasets are English-only. In contrast, current multilingual and multimodal
datasets are either composed of caption-like only or medium-scale or fully
private data. This limits mLLM research for the 7,000 other languages spoken in
the world. We therefore introduce mOSCAR, to the best of our knowledge the
first large-scale multilingual and multimodal document corpus crawled from the
web. It covers 163 languages, 315M documents, 214B tokens and 1.2B images. We
carefully conduct a set of filtering and evaluation steps to make sure mOSCAR
is sufficiently safe, diverse and of good quality. We additionally train two
types of multilingual model to prove the benefits of mOSCAR: (1) a model
trained on a subset of mOSCAR and captioning data and (2) a model train on
captioning data only. The model additionally trained on mOSCAR shows a strong
boost in few-shot learning performance across various multilingual image-text
tasks and benchmarks, confirming previous findings for English-only mLLMs.

Multimodal Large Language Models (mLLMs) that are trained on caption-like and interleaved text-image data, such as mOSCAR, show improved in-context learning capabilities, boost in few-shot learning performance across various multilingual image-text tasks and benchmarks, and address the limitation of current multilingual and multimodal datasets.

mOSCAR：一个大规模的多语言和多模态的文档级语料库

mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus

Transformer-based large language models have displayed impressive in-context
learning capabilities, where a pre-trained model can handle new tasks without
fine-tuning by simply augmenting the query with some input-output examples from
that task. Despite the empirical success, the mechanics of how to train a
Transformer to achieve ICL and the corresponding ICL capacity is mostly elusive
due to the technical challenges of analyzing the nonconvex training problems
resulting from the nonlinear self-attention and nonlinear activation in
Transformers. To the best of our knowledge, this paper provides the first
theoretical analysis of the training dynamics of Transformers with nonlinear
self-attention and nonlinear MLP, together with the ICL generalization
capability of the resulting model. Focusing on a group of binary classification
tasks, we train Transformers using data from a subset of these tasks and
quantify the impact of various factors on the ICL generalization performance on
the remaining unseen tasks with and without data distribution shifts. We also
analyze how different components in the learned Transformers contribute to the
ICL performance. Furthermore, we provide the first theoretical analysis of how
model pruning affects the ICL performance and prove that proper magnitude-based
pruning can have a minimal impact on ICL while reducing inference costs. These
theoretical findings are justified through numerical experiments.

通过理论分析，我们首次探讨了具有非线性自注意力和非线性 MLP 的 Transformer 模型的训练动态和 ICL 泛化能力，重点关注一组二分类任务，研究了各种因素对 ICL 泛化性能的影响，探讨了不同组件对 ICL 性能的贡献，并首次理论分析了模型修剪对 ICL 性能的影响，证明合适的基于大小的修剪可以在降低推理成本的同时对 ICL 产生最小影响，并通过数值实验验证了这些理论结果。

非线性变压器的高效上下文学习训练：理论学习和泛化分析

Training Nonlinear Transformers for Efficient In-Context Learning: A  Theoretical Learning and Generalization Analysis

This paper presents an overview of the PromptCBLUE shared task
(this http URL) held in the CHIP-2023 Conference. This
shared task reformualtes the CBLUE benchmark, and provide a good testbed for
Chinese open-domain or medical-domain large language models (LLMs) in general
medical natural language processing. Two different tracks are held: (a) prompt
tuning track, investigating the multitask prompt tuning of LLMs, (b) probing
the in-context learning capabilities of open-sourced LLMs. Many teams from both
the industry and academia participated in the shared tasks, and the top teams
achieved amazing test results. This paper describes the tasks, the datasets,
evaluation metrics, and the top systems for both tasks. Finally, the paper
summarizes the techniques and results of the evaluation of the various
approaches explored by the participating teams.

该研究论文介绍了 PromptCBLUE 共享任务，该任务重塑了 CBLUE 基准，并为中文开放领域或医学领域的大型语言模型（LLMs）提供了一个良好的测试平台，探索了 LLMs 的多任务提示优化和开源 LLMs 的上下文学习能力调查，并总结了各参与团队在不同方法上的技术和结果评估。