With recent empirical observations, it has been argued that the most
significant aspect of developing accurate language models may be the proper
dataset content and training strategy compared to the number of neural
parameters, training duration or dataset size. Following this argument, we
opted to fine tune a one billion parameter size trained general purpose causal
language model with a dataset curated on team statistics of the Italian
football league first ten game weeks, using low rank adaptation. The limited
training dataset was compiled based on a framework where a powerful commercial
large language model provides distilled paragraphs and question answer pairs as
intended. The training duration was kept relatively short to provide a basis
for our minimal setting exploration. We share our key observations on the
process related to developing a specific purpose language model which is
intended to interpret soccer data with constrained resources in this article.

通过使用低阶适应的方法，我们在基于意大利足球联赛前十轮的球队统计数据上进行了一个十亿参数规模的通用因果语言模型的微调，结果观察到在开发特定目的的语言模型时，数据集内容和训练策略比起神经网络参数数量、训练时长或数据集大小更为重要。

FootGPT：基于简单环境的大型语言模型开发实验

FootGPT : A Large Language Model Development Experiment on a Minimal  Setting

State-of-the-art ASR systems have achieved promising results by modeling
local and global interactions separately. While the former can be computed
efficiently, global interactions are usually modeled via attention mechanisms,
which are expensive for long input sequences. Here, we address this by
extending HyperMixer, an efficient alternative to attention exhibiting linear
complexity, to the Conformer architecture for speech recognition, leading to
HyperConformer. In particular, multi-head HyperConformer achieves comparable or
higher recognition performance while being more efficient than Conformer in
terms of inference speed, memory, parameter count, and available training data.
HyperConformer achieves a word error rate of 2.9% on Librispeech test-clean
with less than 8M neural parameters and a peak memory during training of 5.7GB,
hence trainable with accessible hardware. Encoder speed is between 38% on
mid-length speech and 56% on long speech faster than an equivalent Conformer.
(The HyperConformer recipe is publicly available in:
this https URL)

本文介绍了一种名为 HyperConformer 的语音识别结构，它通过引入高效的 HyperMixer 机制，实现了对于长输入序列较为经济的全局交互建模，并在可获得的训练数据限制下达到与或高于传统结构 Conformer 相似的识别表现。