Imitation learning enables robots to learn and replicate human behavior from
training data. Recent advances in machine learning enable end-to-end learning
approaches that directly process high-dimensional observation data, such as
images. However, these approaches face a critical challenge when processing
data from multiple modalities, inadvertently ignoring data with a lower
correlation to the desired output, especially when using short sampling
periods. This paper presents a useful method to address this challenge, which
amplifies the influence of data with a relatively low correlation to the output
by inputting the data into each neural network layer. The proposed approach
effectively incorporates diverse data sources into the learning process.
Through experiments using a simple pick-and-place operation with raw images and
joint information as input, significant improvements in success rates are
demonstrated even when dealing with data from short sampling periods.

通过在每个神经网络层中输入具有相对较低相关性的数据来增强其对输出的影响，本文提出了一种应对多模态数据挑战的有效方法，成功地将多样的数据源融入到学习过程中，通过使用原始图像和关节信息作为输入的简单拾放操作实验，即使处理来自短采样时期的数据也展示了显著的成功率提高。

神经网络各层输入图像特征的模仿学习

Imitation Learning Inputting Image Feature to Each Layer of Neural  Network

This study addresses generating counterfactual explanations with multimodal
information. Our goal is not only to classify a video into a specific category,
but also to provide explanations on why it is not categorized to a specific
class with combinations of visual-linguistic information. Requirements that the
expected output should satisfy are referred to as counterfactuality in this
paper: (1) Compatibility of visual-linguistic explanations, and (2)
Positiveness/negativeness for the specific positive/negative class. Exploiting
a spatio-temporal region (tube) and an attribute as visual and linguistic
explanations respectively, the explanation model is trained to predict the
counterfactuality for possible combinations of multimodal information in a
post-hoc manner. The optimization problem, which appears during
training/inference, can be efficiently solved by inserting a novel neural
network layer, namely the maximum subpath layer. We demonstrated the
effectiveness of this method by comparison with a baseline of the action
recognition datasets extended for this task. Moreover, we provide
information-theoretical insight into the proposed method.

使用视觉语言信息结合最大子路径神经网络层，实现对于多模式信息组合的反事实解释的有效解决方法，对行为识别数据集进行对比实验并提供了信息论上的解释。

通过预测视频中的反事实性达到多模态解释

Multimodal Explanations by Predicting Counterfactuality in Videos

The iterations of many sparse estimation algorithms are comprised of a fixed
linear filter cascaded with a thresholding nonlinearity, which collectively
resemble a typical neural network layer. Consequently, a lengthy sequence of
algorithm iterations can be viewed as a deep network with shared, hand-crafted
layer weights. It is therefore quite natural to examine the degree to which a
learned network model might act as a viable surrogate for traditional sparse
estimation in domains where ample training data is available. While the
possibility of a reduced computational budget is readily apparent when a
ceiling is imposed on the number of layers, our work primarily focuses on
estimation accuracy. In particular, it is well-known that when a signal
dictionary has coherent columns, as quantified by a large RIP constant, then
most tractable iterative algorithms are unable to find maximally sparse
representations. In contrast, we demonstrate both theoretically and empirically
the potential for a trained deep network to recover minimal $\ell_0$-norm
representations in regimes where existing methods fail. The resulting system is
deployed on a practical photometric stereo estimation problem, where the goal
is to remove sparse outliers that can disrupt the estimation of surface normals
from a 3D scene.

论文研究了稀疏估计算法、神经网络层和深度网络学习在估计准确性方面的应用，展示了深度网络学习可以在稀疏字典具有连贯列时恢复最小 L0 范数表示的潜力，并应用于三维场景表面法线的实际测量问题中。