The Transformer architecture is shown to provide a powerful machine
transduction framework for online handwritten gestures corresponding to glyph
strokes of natural language sentences. The attention mechanism is successfully
used to create latent representations of an end-to-end encoder-decoder model,
solving multi-level segmentation while also learning some language features and
syntax rules. The additional use of a large decoding space with some learned
Byte-Pair-Encoding (BPE) is shown to provide robustness to ablated inputs and
syntax rules. The encoder stack was directly fed with spatio-temporal data
tokens potentially forming an infinitely large input vocabulary, an approach
that finds applications beyond that of this work. Encoder transfer learning
capabilities is also demonstrated on several languages resulting in faster
optimisation and shared parameters. A new supervised dataset of online
handwriting gestures suitable for generic handwriting recognition tasks was
used to successfully train a small transformer model to an average normalised
Levenshtein accuracy of 96% on English or German sentences and 94% in French.

本文中，使用 Transformer 模型结构对在线手写手势符号转化为自然语言句子进行了研究，并展示了该模型的编码器解决多级分割、学习某些语言特征和语法规则等方面的优秀表现。此外，使用学习的 Byte-Pair-Encoding（BPE）和大型解码空间为数据输入和语法规则提供了鲁棒性。研究表明该模型具有迁移学习能力，适用于许多其他语言，并可用于泛用性的手写识别任务。最终，基于一个新的手写手势数据集训练小型 Transformer 模型，对于英语、德语句子平均归一化 Levenshtein 准确率达到 96％，对于法语则为 94％。

使用 Transformer 和自然语言处理进行在线手势识别

Online Gesture Recognition using Transformer and Natural Language  Processing

The Transformer architecture is shown to provide a powerful framework as an
end-to-end model for building expression trees from online handwritten gestures
corresponding to glyph strokes. In particular, the attention mechanism was
successfully used to encode, learn and enforce the underlying syntax of
expressions creating latent representations that are correctly decoded to the
exact mathematical expression tree, providing robustness to ablated inputs and
unseen glyphs. For the first time, the encoder is fed with spatio-temporal data
tokens potentially forming an infinitely large vocabulary, which finds
applications beyond that of online gesture recognition. A new supervised
dataset of online handwriting gestures is provided for training models on
generic handwriting recognition tasks and a new metric is proposed for the
evaluation of the syntactic correctness of the output expression trees. A small
Transformer model suitable for edge inference was successfully trained to an
average normalised Levenshtein accuracy of 94%, resulting in valid postfix RPN
tree representation for 94% of predictions.

该研究使用 Transformer 架构为端到端的在线手写手势建立表达式树提供了强大的框架，并成功利用了注意力机制编码、学习和执行表达式的潜在语法，提供了鲁棒性，并提出了一个新的度量标准用于输出表达式树的语法正确性的评估。