This paper introduces a highly efficient greedy decoding algorithm for
Transducer inference. We propose a novel data structure using CUDA tensors to
represent partial hypotheses in a batch that supports parallelized hypothesis
manipulations. During decoding, our algorithm maximizes GPU parallelism by
adopting a nested-loop design, where the inner loop consumes all blank
predictions, while non-blank predictions are handled in the outer loop. Our
algorithm is general-purpose and can work with both conventional Transducers
and Token-and-Duration Transducers. Experiments show that the label-looping
algorithm can bring a speedup up to 2.0X compared to conventional batched
decoding algorithms when using batch size 32, and can be combined with other
compiler or GPU call-related techniques to bring more speedup. We will
open-source our implementation to benefit the research community.

该论文介绍了一种高效的贪婪解码算法，用于转导推断，通过使用 CUDA 张量表示批量中的部分假设来支持并行化的假设操作，在解码过程中，我们的算法通过采用嵌套循环设计最大化 GPU 并行性，其中内循环处理所有空白预测，而非空白预测在外循环中处理，实验证明，在使用批量大小为 32 时，标签循环算法相较于传统的批量解码算法可以加速多达 2.0 倍，可以与其他编译器或 GPU 调用相关技术结合以获得更多加速，我们将开源我们的实现以造福研究社区。