Existing vision-language models (VLMs) mostly rely on vision encoders to
extract visual features followed by large language models (LLMs) for
visual-language tasks. However, the vision encoders set a strong inductive bias
in abstracting visual representation, e.g., resolution, aspect ratio, and
semantic priors, which could impede the flexibility and efficiency of the VLMs.
Training pure VLMs that accept the seamless vision and language inputs, i.e.,
without vision encoders, remains challenging and rarely explored. Empirical
observations reveal that direct training without encoders results in slow
convergence and large performance gaps. In this work, we bridge the gap between
encoder-based and encoder-free models, and present a simple yet effective
training recipe towards pure VLMs. Specifically, we unveil the key aspects of
training encoder-free VLMs efficiently via thorough experiments: (1) Bridging
vision-language representation inside one unified decoder; (2) Enhancing visual
recognition capability via extra supervision. With these strategies, we launch
EVE, an encoder-free vision-language model that can be trained and forwarded
efficiently. Notably, solely utilizing 35M publicly accessible data, EVE can
impressively rival the encoder-based VLMs of similar capacities across multiple
vision-language benchmarks. It significantly outperforms the counterpart
Fuyu-8B with mysterious training procedures and undisclosed training data. We
believe that EVE provides a transparent and efficient route for developing a
pure decoder-only architecture across modalities. Our code and models are
publicly available at: this https URL

在这项研究中，我们提出了一种简单而有效的训练方法，实现了没有视觉编码器的纯视觉语言模型，并通过桥接视觉语言表示和增强视觉识别能力的策略推出了 EVE 模型，它在多个视觉语言基准测试中显著优于采用类似容量的基于编码器的 VLMs。

揭示无编码器的视觉 - 语言模型

Unveiling Encoder-Free Vision-Language Models

We introduce LongSkywork, a long-context Large Language Model (LLM) capable
of processing up to 200,000 tokens. We provide a training recipe for
efficiently extending context length of LLMs. We identify that the critical
element in enhancing long-context processing capability is to incorporate a
long-context SFT stage following the standard SFT stage. A mere 200 iterations
can convert the standard SFT model into a long-context model. To reduce the
effort in collecting and annotating data for long-context language modeling, we
develop two novel methods for creating synthetic data. These methods are
applied during the continual pretraining phase as well as the Supervised
Fine-Tuning (SFT) phase, greatly enhancing the training efficiency of our
long-context LLMs. Our findings suggest that synthetic long-context SFT data
can surpass the performance of data curated by humans to some extent.
LongSkywork achieves outstanding performance on a variety of long-context
benchmarks. In the Needle test, a benchmark for long-context information
retrieval, our models achieved perfect accuracy across multiple context spans.
Moreover, in realistic application scenarios, LongSkywork-13B demonstrates
performance on par with Claude2.1, the leading long-context model, underscoring
the effectiveness of our proposed methods.

LongSkywork 是一种具有长上下文处理能力的大型语言模型 (LLM)，通过在标准 SFT 阶段之后添加长上下文 SFT 阶段来增强长上下文处理能力，使用合成数据的方法显著提高了训练效率，并在各种长上下文基准测试中取得了出色的表现。

LongSkywork：用于大型语言模型中高效扩展上下文长度的训练方法

LongSkywork: A Training Recipe for Efficiently Extending Context Length  in Large Language Models

Although BERT-style encoder models are heavily used in NLP research, many
researchers do not pretrain their own BERTs from scratch due to the high cost
of training. In the past half-decade since BERT first rose to prominence, many
advances have been made with other transformer architectures and training
configurations that have yet to be systematically incorporated into BERT. Here,
we introduce MosaicBERT, a BERT-style encoder architecture and training recipe
that is empirically optimized for fast pretraining. This efficient architecture
incorporates FlashAttention, Attention with Linear Biases (ALiBi), Gated Linear
Units (GLU), a module to dynamically remove padded tokens, and low precision
LayerNorm into the classic transformer encoder block. The training recipe
includes a 30% masking ratio for the Masked Language Modeling (MLM) objective,
bfloat16 precision, and vocabulary size optimized for GPU throughput, in
addition to best-practices from RoBERTa and other encoder models. When
pretrained from scratch on the C4 dataset, this base model achieves a
downstream average GLUE (dev) score of 79.6 in 1.13 hours on 8 A100 80 GB GPUs
at a cost of roughly $20. We plot extensive accuracy vs. pretraining speed
Pareto curves and show that MosaicBERT base and large are consistently Pareto
optimal when compared to a competitive BERT base and large. This empirical
speed up in pretraining enables researchers and engineers to pretrain custom
BERT-style models at low cost instead of finetune on existing generic models.
We open source our model weights and code.

MosaicBERT 是一种优化用于快速预训练的 BERT 风格编码器架构和训练方法，包括 FlashAttention、ALiBi、GLU、动态去除填充令牌的模块和低精度 LayerNorm 等，基于 C4 数据集，该基础模型在 8 个 A100 80GB GPU 上以 1.13 小时的时间和约 20 美元的成本实现 79.6 的平均 GLUE（dev）得分，相比竞争的 BERT 模型具有稳定的 Pareto 最优性。