As the use of collaborative robots (cobots) in industrial manufacturing
continues to grow, human action recognition for effective human-robot
collaboration becomes increasingly important. This ability is crucial for
cobots to act autonomously and assist in assembly tasks. Recently,
skeleton-based approaches are often used as they tend to generalize better to
different people and environments. However, when processing skeletons alone,
information about the objects a human interacts with is lost. Therefore, we
present a novel approach of integrating object information into skeleton-based
action recognition. We enhance two state-of-the-art methods by treating object
centers as further skeleton joints. Our experiments on the assembly dataset
IKEA ASM show that our approach improves the performance of these
state-of-the-art methods to a large extent when combining skeleton joints with
objects predicted by a state-of-the-art instance segmentation model. Our
research sheds light on the benefits of combining skeleton joints with object
information for human action recognition in assembly tasks. We analyze the
effect of the object detector on the combination for action classification and
discuss the important factors that must be taken into account.

本文探讨了如何将目标物体信息融入基于骨架的行为识别，以提高人员 - 机器人协作的效率和智能化水平，并在 IKEA ASM 数据集的实验中证明了该方法能够提高目前先进方法的性能。

物体信息如何提高基于骨架的人体动作识别在装配任务中的表现

How Object Information Improves Skeleton-based Human Action Recognition  in Assembly Tasks

Human infants acquire their verbal lexicon from minimal prior knowledge of
language based on the statistical properties of phonological distributions and
the co-occurrence of other sensory stimuli. In this study, we propose a novel
fully unsupervised learning method discovering speech units by utilizing
phonological information as a distributional cue and object information as a
co-occurrence cue. The proposed method can not only (1) acquire words and
phonemes from speech signals using unsupervised learning, but can also (2)
utilize object information based on multiple modalities (i.e., vision, tactile,
and auditory) simultaneously. The proposed method is based on the Nonparametric
Bayesian Double Articulation Analyzer (NPB-DAA) discovering phonemes and words
from phonological features, and Multimodal Latent Dirichlet Allocation (MLDA)
categorizing multimodal information obtained from objects. In the experiment,
the proposed method showed higher word discovery performance than the baseline
methods. In particular, words that expressed the characteristics of the object
(i.e., words corresponding to nouns and adjectives) were segmented accurately.
Furthermore, we examined how learning performance is affected by differences in
the importance of linguistic information. When the weight of the word modality
was increased, the performance was further improved compared to the fixed
condition.

通过利用音韵信息和对象信息来发现言语单元的全新无监督学习方法，能够同时根据多种模态（即视觉、触觉和听觉）使用对象信息，该方法基于非参数贝叶斯双连音分析器（NPB-DAA）发现音素和单词以及多模态潜在狄利克雷分配（MLDA）分类来自对象的多模态信息，实验表明，所提出的方法比现有基准方法具有更高的词汇发现性能。

基于双关节分析和共现线索的无监督多模态单词发现

Unsupervised Multimodal Word Discovery based on Double Articulation Analysis with Co-occurrence cues

We address the task of detecting foiled image captions, i.e. identifying
whether a caption contains a word that has been deliberately replaced by a
semantically similar word, thus rendering it inaccurate with respect to the
image being described. Solving this problem should in principle require a
fine-grained understanding of images to detect linguistically valid
perturbations in captions. In such contexts, encoding sufficiently descriptive
image information becomes a key challenge. In this paper, we demonstrate that
it is possible to solve this task using simple, interpretable yet powerful
representations based on explicit object information. Our models achieve
state-of-the-art performance on a standard dataset, with scores exceeding those
achieved by humans on the task. We also measure the upper-bound performance of
our models using gold standard annotations. Our analysis reveals that the
simpler model performs well even without image information, suggesting that the
dataset contains strong linguistic bias.

本文利用基于显式目标信息的简单的可解释且强大的表示方法解决了探测篡改图片标题的任务，并在标准数据集上取得了优于人类的成绩；同时分析揭示了简单模型即使没有图像信息也能取得良好的表现，表明数据集具有强烈的语言倾向。