Our goal is to recognize material categories using images and geometry
information. In many applications, such as construction management, coarse
geometry information is available. We investigate how 3D geometry (surface
normals, camera intrinsic and extrinsic parameters) can be used with 2D
features (texture and color) to improve material classification. We introduce a
new dataset, GeoMat, which is the first to provide both image and geometry data
in the form of: (i) training and testing patches that were extracted at
different scales and perspectives from real world examples of each material
category, and (ii) a large scale construction site scene that includes 160
images and over 800,000 hand labeled 3D points. Our results show that using 2D
and 3D features both jointly and independently to model materials improves
classification accuracy across multiple scales and viewing directions for both
material patches and images of a large scale construction site scene.

本研究旨在以图像和几何信息为基础来识别材料类别。我们提出了一个新的数据集来研究如何使用 3D 几何信息和 2D 特征来提高材料分类的准确性，并且结果表明联合和独立地使用 2D 和 3D 特征来模拟材料可以提高材料分类的准确性。

几何计算辅助的材料识别

Geometry-Informed Material Recognition

Fisher vector (FV) has become a popular image representation. One notable
underlying assumption of the FV framework is that local descriptors are well
decorrelated within each cluster so that the covariance matrix for each
Gaussian can be simplified to be diagonal. Though the FV usually relies on the
Principal Component Analysis (PCA) to decorrelate local features, the PCA is
applied to the entire training data and hence it only diagonalizes the
\textit{universal} covariance matrix, rather than those w.r.t. the local
components. As a result, the local decorrelation assumption is usually not
supported in practice.
To relax this assumption, this paper proposes a completed model of the Fisher
vector, which is termed as the Completed Fisher vector (CFV). The CFV is a more
general framework of the FV, since it encodes not only the variances but also
the correlations of the whitened local descriptors. The CFV thus leads to
improved discriminative power. We take the task of material categorization as
an example and experimentally show that: 1) the CFV outperforms the FV under
all parameter settings; 2) the CFV is robust to the changes in the number of
components in the mixture; 3) even with a relatively small visual vocabulary
the CFV still works well on two challenging datasets.

本文介绍了一种名为 Completed Fisher vector（CFV）的图像表示方法，相较于传统的 Fisher vector（FV），CFV 不仅能够描述局部特征的方差，还能够描述局部特征的相关性，从而提高了鉴别能力，实验结果表明，CFV 在物体分类任务中表现优秀。