In this work, we focus on unsupervised vision-language-action mapping in the
area of robotic manipulation. Recently, multiple approaches employing
pre-trained large language and vision models have been proposed for this task.
However, they are computationally demanding and require careful fine-tuning of
the produced outputs. A more lightweight alternative would be the
implementation of multimodal Variational Autoencoders (VAEs) which can extract
the latent features of the data and integrate them into a joint representation,
as has been demonstrated mostly on image-image or image-text data for the
state-of-the-art models. Here we explore whether and how can multimodal VAEs be
employed in unsupervised robotic manipulation tasks in a simulated environment.
Based on the obtained results, we propose a model-invariant training
alternative that improves the models' performance in a simulator by up to 55%.
Moreover, we systematically evaluate the challenges raised by the individual
tasks such as object or robot position variability, number of distractors or
the task length. Our work thus also sheds light on the potential benefits and
limitations of using the current multimodal VAEs for unsupervised learning of
robotic motion trajectories based on vision and language.

我们关注机器人操作中无监督的视觉 - 语言 - 行为映射，探索多模态变分自编码器在无监督机器人操作任务中的应用，并提出一种模型不变训练方法，成功提高模型在模拟环境中的性能，并对个体任务的挑战进行了系统评估，揭示了当前多模态变分自编码器在基于视觉和语言的无监督机器人运动轨迹学习中的潜在优势和限制。