Transparent objects are encountered frequently in our daily lives, yet
recognizing them poses challenges for conventional vision sensors due to their
unique material properties, not being well perceived from RGB or depth cameras.
Overcoming this limitation, thermal infrared cameras have emerged as a
solution, offering improved visibility and shape information for transparent
objects. In this paper, we present TRansPose, the first large-scale
multispectral dataset that combines stereo RGB-D, thermal infrared (TIR)
images, and object poses to promote transparent object research. The dataset
includes 99 transparent objects, encompassing 43 household items, 27 recyclable
trashes, 29 chemical laboratory equivalents, and 12 non-transparent objects. It
comprises a vast collection of 333,819 images and 4,000,056 annotations,
providing instance-level segmentation masks, ground-truth poses, and completed
depth information. The data was acquired using a FLIR A65 thermal infrared
(TIR) camera, two Intel RealSense L515 RGB-D cameras, and a Franka Emika Panda
robot manipulator. Spanning 87 sequences, TRansPose covers various challenging
real-life scenarios, including objects filled with water, diverse lighting
conditions, heavy clutter, non-transparent or translucent containers, objects
in plastic bags, and multi-stacked objects. TRansPose dataset can be accessed
from the following link: this https URL

本文介绍了 TRansPose 多光谱数据集，该数据集包括 99 种透明物体，其中 43 种家用物品、27 种可回收垃圾、29 种化学实验室用品和 12 种非透明物体，可以提供更好的透明物体视觉识别和形状信息。

TransPose：用于透明物体的大规模多光谱数据集

TRansPose: Large-Scale Multispectral Dataset for Transparent Object

Improving the generalization capabilities of general-purpose robotic agents
has long been a significant challenge actively pursued by research communities.
Existing approaches often rely on collecting large-scale real-world robotic
data, such as the RT-1 dataset. However, these approaches typically suffer from
low efficiency, limiting their capability in open-domain scenarios with new
objects, and diverse backgrounds. In this paper, we propose a novel paradigm
that effectively leverages language-grounded segmentation masks generated by
state-of-the-art foundation models, to address a wide range of pick-and-place
robot manipulation tasks in everyday scenarios. By integrating precise
semantics and geometries conveyed from masks into our multi-view policy model,
our approach can perceive accurate object poses and enable sample-efficient
learning. Besides, such design facilitates effective generalization for
grasping new objects with similar shapes observed during training. Our approach
consists of two distinct steps. First, we introduce a series of foundation
models to accurately ground natural language demands across multiple tasks.
Second, we develop a Multi-modal Multi-view Policy Model that incorporates
inputs such as RGB images, semantic masks, and robot proprioception states to
jointly predict precise and executable robot actions. Extensive real-world
experiments conducted on a Franka Emika robot arm validate the effectiveness of
our proposed paradigm. Real-world demos are shown in YouTube
(this https URL ) and Bilibili
(this https URL ).

本文提出了一种基于语言根据分割掩模的新型范例，来应对日常情况下一系列的抓取和放置机器人操作任务，通过将从掩模中传达的精确语义和几何形状与我们的多视点策略模型相结合，使我们的方法能够感知准确的物体姿态并实现样本高效学习。

铺路把握一切：将基础模型转移，用于通用拾放机器人

Pave the Way to Grasp Anything: Transferring Foundation Models for  Universal Pick-Place Robots

We present a new pipeline for holistic 3D scene understanding from a single
image, which could predict object shapes, object poses, and scene layout. As it
is a highly ill-posed problem, existing methods usually suffer from inaccurate
estimation of both shapes and layout especially for the cluttered scene due to
the heavy occlusion between objects. We propose to utilize the latest deep
implicit representation to solve this challenge. We not only propose an
image-based local structured implicit network to improve the object shape
estimation, but also refine the 3D object pose and scene layout via a novel
implicit scene graph neural network that exploits the implicit local object
features. A novel physical violation loss is also proposed to avoid incorrect
context between objects. Extensive experiments demonstrate that our method
outperforms the state-of-the-art methods in terms of object shape, scene layout
estimation, and 3D object detection.

本文提出了一种基于单幅图像的全面三维场景理解的新管道，该管道可以预测物体形状、物体姿态和场景布局，利用最新的深度隐式表示方法解决了物体形状和布局的不精确估计的问题，通过物理违规损失函数避免了物体之间的错误场景。实验结果表明，该方法在物体形状、场景布局估计和三维物体检测方面优于现有最先进技术。