This study achieved bidirectional translation between descriptions and
actions using small paired data from different modalities. The ability to
mutually generate descriptions and actions is essential for robots to
collaborate with humans in their daily lives, which generally requires a large
dataset that maintains comprehensive pairs of both modality data. However, a
paired dataset is expensive to construct and difficult to collect. To address
this issue, this study proposes a two-stage training method for bidirectional
translation. In the proposed method, we train recurrent autoencoders (RAEs) for
descriptions and actions with a large amount of non-paired data. Then, we
finetune the entire model to bind their intermediate representations using
small paired data. Because the data used for pre-training do not require
pairing, behavior-only data or a large language corpus can be used. We
experimentally evaluated our method using a paired dataset consisting of
motion-captured actions and descriptions. The results showed that our method
performed well, even when the amount of paired data to train was small. The
visualization of the intermediate representations of each RAE showed that
similar actions were encoded in a clustered position and the corresponding
feature vectors were well aligned.

本研究使用少量不同模式的配对数据实现了描述和动作之间的双向翻译，其中使用回归自编码器（RAEs）进行先前训练以处理大量不配对数据，然后用少量配对数据对其进行微调，该方法表现出良好效果，并且可用于大型语言语料库，中间表示的可视化进一步验证了其有效性。