This paper introduces a new technique to measure the feature dependency of
neural network models. The motivation is to better understand a model by
querying whether it is using information from human-understandable features,
e.g., anatomical shape, volume, or image texture. Our method is based on the
principle that if a model is dependent on a feature, then removal of that
feature should significantly harm its performance. A targeted feature is
"removed" by collapsing the dimension in the data distribution that corresponds
to that feature. We perform this by moving data points along the feature
dimension to a baseline feature value while staying on the data manifold, as
estimated by a deep generative model. Then we observe how the model's
performance changes on the modified test data set, with the target feature
dimension removed. We test our method on deep neural network models trained on
synthetic image data with known ground truth, an Alzheimer's disease prediction
task using MRI and hippocampus segmentations from the OASIS-3 dataset, and a
cell nuclei classification task using the Lizard dataset.

通过在数据流形内的特征维度上进行数据点移动，从而在受到影响的测试数据集上观察模型性能的变化，以测量神经网络模型的特征依赖性，并在合成图像数据、阿尔茨海默病预测和细胞核分类任务上测试了该方法。

通过折叠数据流形中的特征维度，衡量神经网络的特征依赖性

Measuring Feature Dependency of Neural Networks by Collapsing Feature  Dimensions in the Data Manifold

We propose a deep neural network for supervised learning on neuroanatomical
shapes. The network directly operates on raw point clouds without the need for
mesh processing or the identification of point correspondences, as spatial
transformer networks map the data to a canonical space. Instead of relying on
hand-crafted shape descriptors, an optimal representation is learned in the
end-to-end training stage of the network. The proposed network consists of
multiple branches, so that features for multiple structures are learned
simultaneously. We demonstrate the performance of our method on two
applications: (i) the prediction of Alzheimer's disease and mild cognitive
impairment and (ii) the regression of the brain age. Finally, we visualize the
important parts of the anatomy for the prediction by adapting the occlusion
method to point clouds.

提出一种基于深度神经网络的神经解剖学形态学的有监督学习方法，直接对原始点云进行操作，通过空间变换网络将数据映射到规范空间，并通过网络的端对端训练阶段学习最优表示，通过多个分支来学习多个结构的特征，应用于阿尔茨海默病和轻度认知障碍的预测和脑龄的回归，最后通过适应点云遮挡法来可视化预测解剖重要部位。