This paper presents a method for learning overcomplete dictionaries composed of two modalities that describe a 3D scene: image intensity and scene depth. We propose a novel Joint Basis Pursuit (JBP) algorithm that finds related sparse features in two modalities using conic programming and integrate it into a two-step dictionary learning algorithm. JBP differs from related convex algorithms because it finds joint sparsity models with different atoms and different coefficient values for intensity and depth. This is crucial for recovering generative models where the same sparse underlying causes (3D features) give rise to different signals (intensity and depth). We give a theoretical bound for the sparse coefficient recovery error obtained by JBP, and show experimentally that JBP is far superior to the state of the art Group Lasso algorithm. When applied to the Middlebury depth-intensity database, our learning algorithm converges to a set of related features, such as pairs of depth and intensity edges or image textures and depth slants. Finally, we show that the learned dictionary and JBP achieve the state of the art depth inpainting performance on time-of-flight 3D data.

本文提出了一种利用两种模态描述三维场景，通过锥形编程找到关联稀疏特征并将其整合到两步字典学习算法中，可恢复具有相同稀疏潜在原因（三维特征）的不同信号（强度和深度），相比最先进的Group Lasso算法，JBP优越很多。同时，该学习算法在时飞3D数据的深度修复性能上达到了最先进水平。

联合强度深度稀疏表示学习