We introduce Falcon2-11B, a foundation model trained on over five trillion tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a vision-to-text model. We report our findings during the training of the Falcon2-11B which follows a multi-stage approach where the early stages are distinguished by their context length and a final stage where we use a curated, high-quality dataset. Additionally, we report the effect of doubling the batch size mid-training and how training loss spikes are affected by the learning rate. The downstream performance of the foundation model is evaluated on established benchmarks, including multilingual and code datasets. The foundation model shows strong generalization across all the tasks which makes it suitable for downstream finetuning use cases. For the vision language model, we report the performance on several benchmarks and show that our model achieves a higher average score compared to open-source models of similar size. The model weights and code of both Falcon2-11B and Falcon2-11B-vlm are made available under a permissive license.

本研究针对训练超五万亿个标记的基础模型Falcon2-11B及其多模态对应模型Falcon2-11B-vlm，提出了多阶段训练的方法，并详细分析了中途加倍批量大小的影响及学习率对训练损失波动的影响。研究结果表明，该基础模型在多项任务上展现出优秀的泛化能力，适合下游微调使用，且相较于同类开源模型，视觉语言模型在多个基准测试中取得了更高的平均分。

Falcon2-11B技术报告