We present two solutions to sentence-level SLR. Sentence-level SLR required
mapping videos of sign language sentences to sequences of gloss labels.
Connectionist Temporal Classification (CTC) has been used as the classifier
level of both models. CTC is used to avoid pre-segmenting the sentences into
individual words. The first model is an LRCN-based model, and the second model
is a Multi-Cue Network. LRCN is a model in which a CNN as a feature extractor
is applied to each frame before feeding them into an LSTM. In the first
approach, no prior knowledge has been leveraged. Raw frames are fed into an
18-layer LRCN with a CTC on top. In the second approach, three main
characteristics (hand shape, hand position, and hand movement information)
associated with each sign have been extracted using Mediapipe. 2D landmarks of
hand shape have been used to create the skeleton of the hands and then are fed
to a CONV-LSTM model. Hand locations and hand positions as relative distance to
head are fed to separate LSTMs. All three sources of information have been then
integrated into a Multi-Cue network with a CTC classification layer. We
evaluated the performance of proposed models on RWTH-PHOENIX-Weather. After
performing an excessive search on model hyper-parameters such as the number of
feature maps, input size, batch size, sequence length, LSTM memory cell,
regularization, and dropout, we were able to achieve 35 Word Error Rate (WER).

本文提出了两种解决方案来解决句子级手语识别问题，并通过使用连接主义时间分类（CTC）作为两种模型的分类器层，介绍了 LRCN-based 模型和 Multi-Cue 网络的工作原理。在评估了 RWTH-PHOENIX-Weather 数据集后，作者通过进行超参数搜索，包括特征图数量、输入大小、批量大小、序列长度、LSTM 内存单元、正则化和失活等，获得了 35% 的字词错误率（WER）。

基于句子级的手语识别框架

Sentence-Level Sign Language Recognition Framework

This paper proposes a do-it-all neural model of human hands, named LISA. The
model can capture accurate hand shape and appearance, generalize to arbitrary
hand subjects, provide dense surface correspondences, be reconstructed from
images in the wild and easily animated. We train LISA by minimizing the shape
and appearance losses on a large set of multi-view RGB image sequences
annotated with coarse 3D poses of the hand skeleton. For a 3D point in the hand
local coordinate, our model predicts the color and the signed distance with
respect to each hand bone independently, and then combines the per-bone
predictions using predicted skinning weights. The shape, color and pose
representations are disentangled by design, allowing to estimate or animate
only selected parameters. We experimentally demonstrate that LISA can
accurately reconstruct a dynamic hand from monocular or multi-view sequences,
achieving a noticeably higher quality of reconstructed hand shapes compared to
baseline approaches. Project page:
this https URL

该研究提出了一种名为 LISA 的全方位神经手模型，该模型可以捕捉准确的手部形状和外观，具有密集的表面对应性，并且易于重建。通过在大量 RGB 图像序列上训练，我们的模型可以预测每个手骨的颜色和距离，并结合预测的皮肤权重。实验表明，LISA 可以从单目或多视角序列准确重建动态手部，与基线方法相比，具有更高质量的重建手部形状。

LISA: 学习手部的隐式形状和外观

LISA: Learning Implicit Shape and Appearance of Hands

In this paper, we strive to answer two questions: What is the current state
of 3D hand pose estimation from depth images? And, what are the next challenges
that need to be tackled? Following the successful Hands In the Million
Challenge (HIM2017), we investigate the top 10 state-of-the-art methods on
three tasks: single frame 3D pose estimation, 3D hand tracking, and hand pose
estimation during object interaction. We analyze the performance of different
CNN structures with regard to hand shape, joint visibility, view point and
articulation distributions. Our findings include: (1) isolated 3D hand pose
estimation achieves low mean errors (10 mm) in the view point range of [70,
120] degrees, but it is far from being solved for extreme view points; (2) 3D
volumetric representations outperform 2D CNNs, better capturing the spatial
structure of the depth data; (3) Discriminative methods still generalize poorly
to unseen hand shapes; (4) While joint occlusions pose a challenge for most
methods, explicit modeling of structure constraints can significantly narrow
the gap between errors on visible and occluded joints.

通过 HIM2017 数据集，分析了当前十大最先进的 3D 手势估计方法在三个任务上的表现，探究了不同的 CNN 结构对手的形状、关节可见度、视角和关节分布的表现，结果表明孤立的 3D 手势估计在视角范围 [70,120] 度时的平均误差较低（10 毫米），但对于极端视角的估计还远未完成。