While GPUs are responsible for training the vast majority of state-of-the-art
deep learning models, the implications of their architecture are often
overlooked when designing new deep learning (DL) models. As a consequence,
modifying a DL model to be more amenable to the target hardware can
significantly improve the runtime performance of DL training and inference. In
this paper, we provide a set of guidelines for users to maximize the runtime
performance of their transformer models. These guidelines have been created by
carefully considering the impact of various model hyperparameters controlling
model shape on the efficiency of the underlying computation kernels executed on
the GPU. We find the throughput of models with efficient model shapes is up to
39\% higher while preserving accuracy compared to models with a similar number
of parameters but with unoptimized shapes.

通过仔细考虑 GPU 上执行的基础计算核心的各种模型超参数对模型形状效率的影响，我们提供了一套指南，以最大化用户的 Transformer 模型的运行时性能。通过优化模型形状，与具有类似参数但形状未经优化的模型相比，高效模型形状的吞吐量提高了多达 39％，同时保持准确性。

与硬件协同设计模型架构的案例

The Case for Co-Designing Model Architectures with Hardware

In this paper, we introduce MCTensor, a library based on PyTorch for
providing general-purpose and high-precision arithmetic for DL training.
MCTensor is used in the same way as PyTorch Tensor: we implement multiple
basic, matrix-level computation operators and NN modules for MCTensor with
identical PyTorch interface. Our algorithms achieve high precision computation
and also benefits from heavily-optimized PyTorch floating-point arithmetic. We
evaluate MCTensor arithmetic against PyTorch native arithmetic for a series of
tasks, where models using MCTensor in float16 would match or outperform the
PyTorch model with float32 or float64 precision.

本文介绍了 MCTensor，这是基于 PyTorch 的库，提供 DL 训练的通用和高精度算术，在相同的 PyTorch Tensor 方式下，我们实现了多个基本的矩阵级计算运算符和 NN 模块，我们评估了 MCTensor 算术与 PyTorch 本地算术进行比较，结果表明，使用 MCTensor 在 float16 下的模型与使用 float32 或 float64 精度的 PyTorch 模型相匹配或超出表现，并获得高精度计算的好处。