What makes generalization hard for imitation learning in visual robotic
manipulation? This question is difficult to approach at face value, but the
environment from the perspective of a robot can often be decomposed into
enumerable factors of variation, such as the lighting conditions or the
placement of the camera. Empirically, generalization to some of these factors
have presented a greater obstacle than others, but existing work sheds little
light on precisely how much each factor contributes to the generalization gap.
Towards an answer to this question, we study imitation learning policies in
simulation and on a real robot language-conditioned manipulation task to
quantify the difficulty of generalization to different (sets of) factors. We
also design a new simulated benchmark of 19 tasks with 11 factors of variation
to facilitate more controlled evaluations of generalization. From our study, we
determine an ordering of factors based on generalization difficulty, that is
consistent across simulation and our real robot setup.

模仿学习中泛化难题的困难源自于视觉机器人操作中的因素变化，作者研究了模拟仿真和实际机器人任务中不同因素的泛化难度，并设计了一个新的基准测试用来评估泛化效果。

分解视觉机器人操作中的模仿学习普遍化差距

Decomposing the Generalization Gap in Imitation Learning for Visual  Robotic Manipulation

Unsupervised disentanglement has been shown to be theoretically impossible
without inductive biases on the models and the data. As an alternative
approach, recent methods rely on limited supervision to disentangle the factors
of variation and allow their identifiability. While annotating the true
generative factors is only required for a limited number of observations, we
argue that it is infeasible to enumerate all the factors of variation that
describe a real-world image distribution. To this end, we propose a method for
disentangling a set of factors which are only partially labeled, as well as
separating the complementary set of residual factors that are never explicitly
specified. Our success in this challenging setting, demonstrated on synthetic
benchmarks, gives rise to leveraging off-the-shelf image descriptors to
partially annotate a subset of attributes in real image domains (e.g. of human
faces) with minimal manual effort. Specifically, we use a recent language-image
embedding model (CLIP) to annotate a set of attributes of interest in a
zero-shot manner and demonstrate state-of-the-art disentangled image
manipulation results.

通过有限的监督训练，结合图像描述符技术（CLIP），在真实图像数据中标注和区分不同的变化因素，从而实现图像的去操纵处理。

一幅图片胜过千言万语：朝着野外图像解缠趋势迈进

An Image is Worth More Than a Thousand Words: Towards Disentanglement in  the Wild

Disentangling factors of variation has become a very challenging problem on
representation learning. Existing algorithms suffer from many limitations, such
as unpredictable disentangling factors, poor quality of generated images from
encodings, lack of identity information, etc. In this paper, we propose a
supervised learning model called DNA-GAN which tries to disentangle different
factors or attributes of images. The latent representations of images are
DNA-like, in which each individual piece (of the encoding) represents an
independent factor of the variation. By annihilating the recessive piece and
swapping a certain piece of one latent representation with that of the other
one, we obtain two different representations which could be decoded into two
kinds of images with the existence of the corresponding attribute being
changed. In order to obtain realistic images and also disentangled
representations, we further introduce the discriminator for adversarial
training. Experiments on Multi-PIE and CelebA datasets finally demonstrate that
our proposed method is effective for factors disentangling and even overcome
certain limitations of the existing methods.

本文提出了一种名为 DNA-GAN 的监督学习模型，其通过将图像的不同特征或属性分解开来，达到了逼真的图像生成和特征分离的效果，并在 Multi-PIE 和 CelebA 数据集上得到实验证明其在因素分离方面是有效的。

DNA-GAN：从多属性图像中学习解缠表示

DNA-GAN: Learning Disentangled Representations from Multi-Attribute  Images

Deep learning has enjoyed a great deal of success because of its ability to
learn useful features for tasks such as classification. But there has been less
exploration in learning the factors of variation apart from the classification
signal. By augmenting autoencoders with simple regularization terms during
training, we demonstrate that standard deep architectures can discover and
explicitly represent factors of variation beyond those relevant for
categorization. We introduce a cross-covariance penalty (XCov) as a method to
disentangle factors like handwriting style for digits and subject identity in
faces. We demonstrate this on the MNIST handwritten digit database, the Toronto
Faces Database (TFD) and the Multi-PIE dataset by generating manipulated
instances of the data. Furthermore, we demonstrate these deep networks can
extrapolate `hidden' variation in the supervised signal.

本研究使用正则化方法增强自编码器，展示了深度学习架构可以识别和表达除分类信号以外的变化因素，并通过引入交叉协方差惩罚（XCov）方法，证明了可以分离出手写数字风格和面部主体身份等因素。研究在 MNIST 手写数字数据库，Toronto 人脸数据库和 Multi-PIE 数据集上进行，并演示了这些深度网络可以推广监督信号中的 ` 隐藏 ' 变化。