We study recent research advances that improve large language models through
efficient pre-training and scaling, and open datasets and tools. We combine
these advances to introduce Cerebras-GPT, a family of open compute-optimal
language models scaled from 111M to 13B parameters. We train Cerebras-GPT
models on the Eleuther Pile dataset following DeepMind Chinchilla scaling rules
for efficient pre-training (highest accuracy for a given compute budget). We
characterize the predictable power-law scaling and compare Cerebras-GPT with
other publicly-available models to show all Cerebras-GPT models have
state-of-the-art training efficiency on both pre-training and downstream
objectives. We describe our learnings including how Maximal Update
Parameterization ($\mu$P) can further improve large model scaling, improving
accuracy and hyperparameter predictability at scale. We release our pre-trained
models and code, making this paper the first open and reproducible work
comparing compute-optimal model scaling to models trained on fixed dataset
sizes. Cerebras-GPT models are available on HuggingFace:
this https URL

通过高效的预训练和扩展、开放数据集和工具来改善大语言模型的最近研究进展；我们将这些进展结合起来，推出了 Cerebras-GPT，这是一个从 111M 扩展到 13B 参数的开放计算最优语言模型家族；我们在 Eleuther Pile 数据集上训练 Cerebras-GPT 模型，并使用 DeepMind Chinchilla 规则进行高效预训练，比较 Cerebras-GPT 和其他公开可用模型，以展示 Cerebras-GPT 上所有模型均在预训练和下游目标上具有最先进的训练效率。我们描述了我们的经验教训，包括最大更新参数化（μP）如何进一步改善大型模型扩展，在规模上提高精度和超参数预测可预测性；我们发布了我们的预训练模型和代码，使本文成为首个将计算最优模型缩放与基于固定数据集尺寸训练的模型进行比较的开放和可重复使用的工作。

Cerebras-GPT：基于 Cerebras 晶片集群训练的优化开放式云算模型

Cerebras-GPT: Open Compute-Optimal Language Models Trained on the  Cerebras Wafer-Scale Cluster

We present a simple yet effective end-to-end Video-language Pre-training
(VidLP) framework, Masked Contrastive Video-language Pretraining (MAC), for
video-text retrieval tasks. Our MAC aims to reduce video representation's
spatial and temporal redundancy in the VidLP model by a mask sampling mechanism
to improve pre-training efficiency. Comparing conventional temporal sparse
sampling, we propose to randomly mask a high ratio of spatial regions and only
feed visible regions into the encoder as sparse spatial sampling. Similarly, we
adopt the mask sampling technique for text inputs for consistency. Instead of
blindly applying the mask-then-prediction paradigm from MAE, we propose a
masked-then-alignment paradigm for efficient video-text alignment. The
motivation is that video-text retrieval tasks rely on high-level alignment
rather than low-level reconstruction, and multimodal alignment with masked
modeling encourages the model to learn a robust and general multimodal
representation from incomplete and unstable inputs. Coupling these designs
enables efficient end-to-end pre-training: reduce FLOPs (60% off), accelerate
pre-training (by 3x), and improve performance. Our MAC achieves
state-of-the-art results on various video-text retrieval datasets, including
MSR-VTT, DiDeMo, and ActivityNet. Our approach is omnivorous to input
modalities. With minimal modifications, we achieve competitive results on
image-text retrieval tasks.

本文提出了一种简单而有效的视频 - 语言预训练框架，用于视频 - 文本检索任务，并采用掩蔽对比视频 - 语言预训练进行了优化，以提高预训练效率和性能，并实现了与图像 - 文本检索任务的竞争结果。