Large Language Models (LLMs) have achieved remarkable results, but their
increasing resource demand has become a major obstacle to the development of
powerful and accessible super-human intelligence. This report introduces
JetMoE-8B, a new LLM trained with less than $0.1 million, using 1.25T tokens
from carefully mixed open-source corpora and 30,000 H100 GPU hours. Despite its
low cost, the JetMoE-8B demonstrates impressive performance, with JetMoE-8B
outperforming the Llama2-7B model and JetMoE-8B-Chat surpassing the
Llama2-13B-Chat model. These results suggest that LLM training can be much more
cost-effective than generally thought. JetMoE-8B is based on an efficient
Sparsely-gated Mixture-of-Experts (SMoE) architecture, composed of attention
and feedforward experts. Both layers are sparsely activated, allowing JetMoE-8B
to have 8B parameters while only activating 2B for each input token, reducing
inference computation by about 70% compared to Llama2-7B. Moreover, JetMoE-8B
is highly open and academia-friendly, using only public datasets and training
code. All training parameters and data mixtures have been detailed in this
report to facilitate future efforts in the development of open foundation
models. This transparency aims to encourage collaboration and further
advancements in the field of accessible and efficient LLMs. The model weights
are publicly available at this https URL

JetMoE-8B 是一种高性价比、透明和学术友好的基于 Sparsely-gated Mixture-of-Experts (SMoE) 架构的大型语言模型，仅需要不足 10 万美元的培训成本，拥有 8B 个参数，使用公共数据集和训练代码，能在降低推理计算约 70% 的情况下实现令人瞩目的性能表现，为开放的基础模型的发展提供了透明而鼓励合作的方式。

JetMoE：以 0.1M 美元达到 Llama2 性能

JetMoE: Reaching Llama2 Performance with 0.1M Dollars

Foundation models are powerful technologies: how they are released publicly
directly shapes their societal impact. In this position paper, we focus on open
foundation models, defined here as those with broadly available model weights
(e.g. Llama 2, Stable Diffusion XL). We identify five distinctive properties
(e.g. greater customizability, poor monitoring) of open foundation models that
lead to both their benefits and risks. Open foundation models present
significant benefits, with some caveats, that span innovation, competition, the
distribution of decision-making power, and transparency. To understand their
risks of misuse, we design a risk assessment framework for analyzing their
marginal risk. Across several misuse vectors (e.g. cyberattacks, bioweapons),
we find that current research is insufficient to effectively characterize the
marginal risk of open foundation models relative to pre-existing technologies.
The framework helps explain why the marginal risk is low in some cases,
clarifies disagreements about misuse risks by revealing that past work has
focused on different subsets of the framework with different assumptions, and
articulates a way forward for more constructive debate. Overall, our work helps
support a more grounded assessment of the societal impact of open foundation
models by outlining what research is needed to empirically validate their
theoretical benefits and risks.

开放基础模型具有创新、竞争、决策权分配和透明度等显著优势，但当前的研究还不足以有效地表征相对于现有技术的边际风险，因此需要进一步的研究来实证验证其理论上的利益和风险。

开放基础模型的社会影响

On the Societal Impact of Open Foundation Models

Recent studies have advocated for fully open foundation models to promote
transparency and open science. As an initial step, the Open Whisper-style
Speech Model (OWSM) reproduced OpenAI's Whisper using publicly available data
and open-source toolkits. With the aim of reproducing Whisper, the previous
OWSM v1 through v3 models were still based on Transformer, which might lead to
inferior performance compared to other state-of-the-art speech encoders. In
this work, we aim to improve the performance and efficiency of OWSM without
extra training data. We present E-Branchformer based OWSM v3.1 models at two
scales, i.e., 100M and 1B. The 1B model is the largest E-Branchformer based
speech model that has been made publicly available. It outperforms the previous
OWSM v3 in a vast majority of evaluation benchmarks, while demonstrating up to
25% faster inference speed. We publicly release the data preparation scripts,
pre-trained models and training logs.

利用 E-Branchformer 提高 OWSM v3.1 模型的性能和效率，尤其在推理速度上具有 25% 的提升，并推出具有 1B 规模的最大 E-Branchformer 语音模型。