Equipping multi-fingered robots with tactile sensing is crucial for achieving
the precise, contact-rich, and dexterous manipulation that humans excel at.
However, relying solely on tactile sensing fails to provide adequate cues for
reasoning about objects' spatial configurations, limiting the ability to
correct errors and adapt to changing situations. In this paper, we present
Tactile Adaptation from Visual Incentives (TAVI), a new framework that enhances
tactile-based dexterity by optimizing dexterous policies using vision-based
rewards. First, we use a contrastive-based objective to learn visual
representations. Next, we construct a reward function using these visual
representations through optimal-transport based matching on one human
demonstration. Finally, we use online reinforcement learning on our robot to
optimize tactile-based policies that maximize the visual reward. On six
challenging tasks, such as peg pick-and-place, unstacking bowls, and flipping
slender objects, TAVI achieves a success rate of 73% using our four-fingered
Allegro robot hand. The increase in performance is 108% higher than policies
using tactile and vision-based rewards and 135% higher than policies without
tactile observational input. Robot videos are best viewed on our project
website: this https URL

通过利用视觉奖励来优化敏感性策略的触觉适应 (TAVI) 框架，通过在线强化学习，在多指机器人上取得了可观的成功率，提高了性能。