Rising computational demands of modern natural language processing (NLP)
systems have increased the barrier to entry for cutting-edge research while
posing serious environmental concerns. Yet, progress on model efficiency has
been impeded by practical challenges in model evaluation and comparison. For
example, hardware is challenging to control due to disparate levels of
accessibility across different institutions. Moreover, improvements in metrics
such as FLOPs often fail to translate to progress in real-world applications.
In response, we introduce Pentathlon, a benchmark for holistic and realistic
evaluation of model efficiency. Pentathlon focuses on inference, which accounts
for a majority of the compute in a model's lifecycle. It offers a
strictly-controlled hardware platform, and is designed to mirror real-world
applications scenarios. It incorporates a suite of metrics that target
different aspects of efficiency, including latency, throughput, memory
overhead, and energy consumption. Pentathlon also comes with a software library
that can be seamlessly integrated into any codebase and enable evaluation. As a
standardized and centralized evaluation platform, Pentathlon can drastically
reduce the workload to make fair and reproducible efficiency comparisons. While
initially focused on natural language processing (NLP) models, Pentathlon is
designed to allow flexible extension to other fields. We envision Pentathlon
will stimulate algorithmic innovations in building efficient models, and foster
an increased awareness of the social and environmental implications in the
development of future-generation NLP models.

现代自然语言处理（NLP）系统的计算需求增加了最前沿研究的门槛，同时引发了严重的环境问题。然而，在模型效率方面取得进展却受到了模型评估和比较中的实际挑战的阻碍。为了应对这一问题，本文介绍了 Pentathlon，一个用于模型效率全面和现实评估的基准测试。Pentathlon 主要关注推理，并提供一个严格控制的硬件平台，旨在模拟真实的应用场景。它结合了一套针对效率不同方面的度量标准，包括延迟、吞吐量、内存开销和能源消耗。Pentathlon 还提供一个软件库，可以无缝集成到任何代码库中，用于评估。作为一个标准化和集中化的评估平台，Pentathlon 可以大大减轻工作量，实现公平且可重现的效率比较。尽管最初专注于自然语言处理（NLP）模型，Pentathlon 的设计允许灵活扩展到其他领域。我们预计 Pentathlon 将促进建立高效模型的算法创新，并增加对未来 NLP 模型发展中的社会和环境影响的关注。

效率十项全能：效率评估的标准竞技场

Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation

We present the first unified study of the efficiency of self-attention-based
Transformer variants spanning text, speech and vision. We identify input length
thresholds (tipping points) at which efficient Transformer variants become more
efficient than vanilla models, using a variety of efficiency metrics (latency,
throughput, and memory). To conduct this analysis for speech, we introduce
L-HuBERT, a novel local-attention variant of a self-supervised speech model. We
observe that these thresholds are (a) much higher than typical dataset sequence
lengths and (b) dependent on the metric and modality, showing that choosing the
right model depends on modality, task type (long-form vs. typical context) and
resource constraints (time vs. memory). By visualising the breakdown of the
computational costs for transformer components, we also show that
non-self-attention components exhibit significant computational costs. We
release our profiling toolkit at
this https URL .

本文综合了文本、语音和视觉领域中基于自注意力的 Transformer 变体的效率研究，通过使用各种效率指标（延迟、吞吐量和内存），确定了比普通模型更有效的高阈值的输入长度，同时也引入了一种名为 L-HuBERT 的自监督语音模型的本地关注变体以在语音领域进行分析。

何时使用高效自注意力？对文本、语音和图像变量进行分析的 Transformer 变量

When to Use Efficient Self Attention? Profiling Text, Speech and Image  Transformer Variants

Efficient and compact neural network models are essential for enabling the
deployment on mobile and embedded devices. In this work, we point out that
typical design metrics for gauging the efficiency of neural network
architectures -- total number of operations and parameters -- are not
sufficient. These metrics may not accurately correlate with the actual
deployment metrics such as energy and memory footprint. We show that throughput
and energy varies by up to 5X across different neural network operation types
on an off-the-shelf Arm Cortex-M7 microcontroller. Furthermore, we show that
the memory required for activation data also need to be considered, apart from
the model parameters, for network architecture exploration studies.

本研究指出评估神经网络架构效率的典型指标 -- 操作总数和参数总数 -- 并不足够准确，还需要考虑吞吐量和能量消耗等实际部署指标，同时需要将激活数据所需的内存也纳入考虑。实证结果表明，在标准微控制器上，不同神经网络操作类型的吞吐量和能量消耗相差高达 5 倍。