Numerous sign language datasets exist, yet they typically cover only a limited selection of the thousands of signs used globally. Moreover, creating diverse sign language datasets is an expensive and challenging task due to the costs associated with gathering a varied group of signers. Motivated by these challenges, we aimed to develop a solution that addresses these limitations. In this context, we focused on textually describing body movements from skeleton keypoint sequences, leading to the creation of a new dataset. We structured this dataset around AUTSL, a comprehensive isolated Turkish sign language dataset. We also developed a baseline model, SkelCap, which can generate textual descriptions of body movements. This model processes the skeleton keypoints data as a vector, applies a fully connected layer for embedding, and utilizes a transformer neural network for sequence-to-sequence modeling. We conducted extensive evaluations of our model, including signer-agnostic and sign-agnostic assessments. The model achieved promising results, with a ROUGE-L score of 0.98 and a BLEU-4 score of 0.94 in the signer-agnostic evaluation. The dataset we have prepared, namely the AUTSL-SkelCap, will be made publicly available soon.

我们开发了一个新的数据集AUTSL-SkelCap，该数据集以土耳其独立手语(AUTSL)为基础，通过对骨骼关键点序列的文字描述来生成。我们还开发了一个基线模型SkelCap，用于生成身体动作的文字描述。通过对模型进行广泛的评价，包括对签名者无关和手势无关的评估，我们获得了令人满意的结果，签名者无关评估中的ROUGE-L得分为0.98，BLEU-4得分为0.94。我们准备好的数据集AUTSL-SkelCap将很快公开发布。

SkelCap: 从骨架关键点序列自动生成描述文本