Large language models (LLMs) have become a significant workload since their
appearance. However, they are also computationally expensive as they have
billions of parameters and are trained with massive amounts of data. Thus,
recent works have developed dedicated CUDA kernels for LLM training and
inference instead of relying on compilergenerated ones, so that hardware
resources are as fully utilized as possible. In this work, we explore the
possibility of GPU native instruction optimization to further push the CUDA
kernels to extreme performance. Contrary to prior works, we adopt an automatic
optimization approach by defining a search space of possible GPU native
instruction schedules, and then we apply stochastic search to perform
optimization. Experiments show that SIP can further improve CUDA kernel
throughput by automatically discovering better GPU native instruction schedules
and the optimized schedules are tested by 10 million test samples.

通过自动优化搜索空间和随机搜索方法，我们实现了 GPU 原生指令优化，从而进一步提高了 CUDA 核心的吞吐量，并通过 1000 万个测试样本对优化后的指令进行了测试。

SIP: 通过随机指令摄动自动调优 GPU 本地调度

SIP: Autotuning GPU Native Schedules via Stochastic Instruction  Perturbation

We introduce QUICK, a group of novel optimized CUDA kernels for the efficient
inference of quantized Large Language Models (LLMs). QUICK addresses the shared
memory bank-conflict problem of state-of-the-art mixed precision matrix
multiplication kernels. Our method interleaves the quantized weight matrices of
LLMs offline to skip the shared memory write-back after the dequantization. We
demonstrate up to 1.91x speedup over existing kernels of AutoAWQ on larger
batches and up to 1.94x throughput gain on representative LLM models on various
NVIDIA GPU devices.

QUICK 是一组新型优化的 CUDA 内核，用于高效推理量化的大型语言模型。QUICK 解决了现有混合精度矩阵乘法内核中的共享内存冲突问题，通过离线交错量化的权重矩阵跳过解量化后的共享内存写回。我们展示了在较大批次上相对于 AutoAWQ 现有内核的高达 1.91 倍的加速效果，并在各种 NVIDIA GPU 设备上相对于代表性 LLM 模型获得高达 1.94 倍的吞吐量提升。

快速：量化感知的交错和无冲突内核用于高效 LLM 推理

QUICK: Quantization-aware Interleaving and Conflict-free Kernel for  efficient LLM inference

Neural graphics primitives, parameterized by fully connected neural networks,
can be costly to train and evaluate. We reduce this cost with a versatile new
input encoding that permits the use of a smaller network without sacrificing
quality, thus significantly reducing the number of floating point and memory
access operations: a small neural network is augmented by a multiresolution
hash table of trainable feature vectors whose values are optimized through
stochastic gradient descent. The multiresolution structure allows the network
to disambiguate hash collisions, making for a simple architecture that is
trivial to parallelize on modern GPUs. We leverage this parallelism by
implementing the whole system using fully-fused CUDA kernels with a focus on
minimizing wasted bandwidth and compute operations. We achieve a combined
speedup of several orders of magnitude, enabling training of high-quality
neural graphics primitives in a matter of seconds, and rendering in tens of
milliseconds at a resolution of ${1920\!\times\!1080}$.

通过多分辨率哈希表和 CUDA 核心的并行处理，我们能够在几秒钟内训练和呈现高品质的神经图形原语，极大地减少了浮点运算和内存访问操作。

多分辨率哈希编码的即时神经图形基元

Instant Neural Graphics Primitives with a Multiresolution Hash Encoding

We introduce PyTorch Geometric, a library for deep learning on irregularly
structured input data such as graphs, point clouds and manifolds, built upon
PyTorch. In addition to general graph data structures and processing methods,
it contains a variety of recently published methods from the domains of
relational learning and 3D data processing. PyTorch Geometric achieves high
data throughput by leveraging sparse GPU acceleration, by providing dedicated
CUDA kernels and by introducing efficient mini-batch handling for input
examples of different size. In this work, we present the library in detail and
perform a comprehensive comparative study of the implemented methods in
homogeneous evaluation scenarios.

PyTorch Geometric 是一个基于 PyTorch 的深度学习库，用于处理不规则结构的输入数据，如图形，点云和流形。该库通过利用稀疏 GPU 加速，提供专用的 CUDA 内核以及引入对不同大小的输入示例的高效小批量处理，实现了高数据吞吐量。本文详细介绍了该库，并在同质评估场景下对实现的方法进行了全面比较研究。