We propose V2CNet, a new deep learning framework to automatically translate
the demonstration videos to commands that can be directly used in robotic
applications. Our V2CNet has two branches and aims at understanding the
demonstration video in a fine-grained manner. The first branch has the
encoder-decoder architecture to encode the visual features and sequentially
generate the output words as a command, while the second branch uses a Temporal
Convolutional Network (TCN) to learn the fine-grained actions. By jointly
training both branches, the network is able to model the sequential information
of the command, while effectively encodes the fine-grained actions. The
experimental results on our new large-scale dataset show that V2CNet
outperforms recent state-of-the-art methods by a substantial margin, while its
output can be applied in real robotic applications. The source code and trained
models will be made available.

该研究提出了一种新的深度学习框架 V2CNet，用于自动将演示视频翻译成可直接用于机器人应用的命令。这个网络具有两个分支，分别用于编码视觉特征、序列化生成命令和学习精细行动。研究结果表明，V2CNet 在大规模数据集上的性能优于最新的其他方法，且其输出可以应用于实际机器人应用中。