To the best of our knowledge, there are no methods today for training
differentially private regression models on sparse input data. To remedy this,
we adapt the Frank-Wolfe algorithm for $L_1$ penalized linear regression to be
aware of sparse inputs and to use them effectively. In doing so, we reduce the
training time of the algorithm from $\mathcal{O}( T D S + T N S)$ to
$\mathcal{O}(N S + T \sqrt{D} \log{D} + T S^2)$, where $T$ is the number of
iterations and a sparsity rate $S$ of a dataset with $N$ rows and $D$ features.
Our results demonstrate that this procedure can reduce runtime by a factor of
up to $2,200\times$, depending on the value of the privacy parameter $\epsilon$
and the sparsity of the dataset.

利用 Frank-Wolfe 算法对稀疏输入数据训练差分隐私回归模型，大幅减少了训练时间，可以提升多达 2200 倍的运行速度。

通过更快的 Frank-Wolfe 迭代来扩展差分隐私 LASSO 正则化逻辑回归

Scaling Up Differentially Private LASSO Regularized Logistic Regression  via Faster Frank-Wolfe Iterations

Generally, convolutional neural networks (CNNs) process data on a regular
grid, e.g. data generated by ordinary cameras. Designing CNNs for sparse and
irregularly spaced input data is still an open research problem with numerous
applications in autonomous driving, robotics, and surveillance. In this paper,
we propose an algebraically-constrained normalized convolution layer for CNNs
with highly sparse input that has a smaller number of network parameters
compared to related work. We propose novel strategies for determining the
confidence from the convolution operation and propagating it to consecutive
layers. We also propose an objective function that simultaneously minimizes the
data error while maximizing the output confidence. To integrate structural
information, we also investigate fusion strategies to combine depth and RGB
information in our normalized convolution network framework. In addition, we
introduce the use of output confidence as an auxiliary information to improve
the results. The capabilities of our normalized convolution network framework
are demonstrated for the problem of scene depth completion. Comprehensive
experiments are performed on the KITTI-Depth and the NYU-Depth-v2 datasets. The
results clearly demonstrate that the proposed approach achieves superior
performance while requiring only about 1-5% of the number of parameters
compared to the state-of-the-art methods.

本文提出了一种代数约束归一化卷积层，可以用于处理高度稀疏的输入数据，结合深度和 RGB 信息，同时最小化数据误差和最大化输出置信度，解决场景深度补全问题，具有参数效率等优点。

通过 CNN 进行置信度传递，用于引导稀疏深度回归

Confidence Propagation through CNNs for Guided Sparse Depth Regression

In most computer vision applications, convolutional neural networks (CNNs)
operate on dense image data generated by ordinary cameras. Designing CNNs for
sparse and irregularly spaced input data is still an open problem with numerous
applications in autonomous driving, robotics, and surveillance. To tackle this
challenging problem, we introduce an algebraically-constrained convolution
layer for CNNs with sparse input and demonstrate its capabilities for the scene
depth completion task. We propose novel strategies for determining the
confidence from the convolution operation and propagating it to consecutive
layers. Furthermore, we propose an objective function that simultaneously
minimizes the data error while maximizing the output confidence. Comprehensive
experiments are performed on the KITTI depth benchmark and the results clearly
demonstrate that the proposed approach achieves superior performance while
requiring three times fewer parameters than the state-of-the-art methods.
Moreover, our approach produces a continuous pixel-wise confidence map enabling
information fusion, state inference, and decision support.

本研究提出一种代数约束卷积层来解决在卷积神经网络中处理稀疏和不规则间隔输入数据的问题，并且在场景深度完成任务中展示其能力，通过新颖的方法确定置信度并传递到后续层，并提出一个目标函数，同时最小化数据误差并最大化输出置信度，在 KITTI 深度基准上进行了广泛实验并证明了明显的优越性。

基于 CNN 的稀疏数据回归置信度传播

Propagating Confidences through CNNs for Sparse Data Regression

Applying machine learning techniques to the quickly growing data in science
and industry requires highly-scalable algorithms. Large datasets are most
commonly processed "data parallel" distributed across many nodes. Each node's
contribution to the overall gradient is summed using a global allreduce. This
allreduce is the single communication and thus scalability bottleneck for most
machine learning workloads. We observe that frequently, many gradient values
are (close to) zero, leading to sparse of sparsifyable communications. To
exploit this insight, we analyze, design, and implement a set of
communication-efficient protocols for sparse input data, in conjunction with
efficient machine learning algorithms which can leverage these primitives. Our
communication protocols generalize standard collective operations, by allowing
processes to contribute arbitrary sparse input data vectors. Our generic
communication library, SparCML, extends MPI to support additional features,
such as non-blocking (asynchronous) operations and low-precision data
representations. As such, SparCML and its techniques will form the basis of
future highly-scalable machine learning frameworks.

运用稀疏输入数据设计通信协议，使得机器学习与 MPI 协议衔接，从而实现高可扩展性。