Deep Neural Networks (DNNs) have been a large driver and enabler for AI
breakthroughs in recent years. These models have been getting larger in their
attempt to become more accurate and tackle new upcoming use-cases, including
AR/VR and intelligent assistants. However, the training process of such large
models is a costly and time-consuming process, which typically yields a single
model to fit all targets. To mitigate this, various techniques have been
proposed in the literature, including pruning, sparsification or quantization
of the model weights and updates. While able to achieve high compression rates,
they often incur computational overheads or accuracy penalties. Alternatively,
factorization methods have been leveraged to incorporate low-rank compression
in the training process. Similarly, such techniques (e.g.,~SVD) frequently rely
on the computationally expensive decomposition of layers and are potentially
sub-optimal for non-linear models, such as DNNs. In this work, we take a
further step in designing efficient low-rank models and propose Maestro, a
framework for trainable low-rank layers. Instead of regularly applying a priori
decompositions such as SVD, the low-rank structure is built into the training
process through a generalized variant of Ordered Dropout. This method imposes
an importance ordering via sampling on the decomposed DNN structure. Our
theoretical analysis demonstrates that our method recovers the SVD
decomposition of linear mapping on uniformly distributed data and PCA for
linear autoencoders. We further apply our technique on DNNs and empirically
illustrate that Maestro enables the extraction of lower footprint models that
preserve model performance while allowing for graceful accuracy-latency
tradeoff for the deployment to devices of different capabilities.

设计高效的低秩模型，通过一种广义的 Ordered Dropout 方法将低秩结构内嵌到训练过程中，实现对深度神经网络进行可训练的低秩层插入，从而提取出足够小的模型，保持模型性能并允许在不同性能设备上进行准确性和延迟的权衡。

Maestro: 通过可训练分解揭示低秩结构

Maestro: Uncovering Low-Rank Structures via Trainable Decomposition

We introduce a novel class of sample-based explanations we term
high-dimensional representers, that can be used to explain the predictions of a
regularized high-dimensional model in terms of importance weights for each of
the training samples. Our workhorse is a novel representer theorem for general
regularized high-dimensional models, which decomposes the model prediction in
terms of contributions from each of the training samples: with positive
(negative) values corresponding to positive (negative) impact training samples
to the model's prediction. We derive consequences for the canonical instances
of $\ell_1$ regularized sparse models, and nuclear norm regularized low-rank
models. As a case study, we further investigate the application of low-rank
models in the context of collaborative filtering, where we instantiate
high-dimensional representers for specific popular classes of models. Finally,
we study the empirical performance of our proposed methods on three real-world
binary classification datasets and two recommender system datasets. We also
showcase the utility of high-dimensional representers in explaining model
recommendations.

本研究提出了一种名为高维表现者的新型基于样本的解释方法，可用于解释正则化高维模型的预测结果；同时也进一步研究了低秩模型在协作过滤中的应用，并对三个二分类数据集和两个推荐系统数据集的实证表现进行了研究。

规范化高维模型的表示点选择

Representer Point Selection for Explaining Regularized High-dimensional  Models

We study the problem of learning mixtures of low-rank models, i.e.
reconstructing multiple low-rank matrices from unlabelled linear measurements
of each. This problem enriches two widely studied settings -- low-rank matrix
sensing and mixed linear regression -- by bringing latent variables (i.e.
unknown labels) and structural priors (i.e. low-rank structures) into
consideration. To cope with the non-convexity issues arising from unlabelled
heterogeneous data and low-complexity structure, we develop a three-stage
meta-algorithm that is guaranteed to recover the unknown matrices with
near-optimal sample and computational complexities under Gaussian designs. In
addition, the proposed algorithm is provably stable against random noise. We
complement the theoretical studies with empirical evidence that confirms the
efficacy of our algorithm.

本文研究学习低秩模型的问题， 提出了一种三阶段元算法来解决未标记的异构数据和低复杂度结构带来的非凸性问题，并且可证明其稳定性。

学习低秩模型混合物

Learning Mixtures of Low-Rank Models

The field of medical image reconstruction has seen roughly four types of
methods. The first type tended to be analytical methods, such as filtered
back-projection (FBP) for X-ray computed tomography (CT) and the inverse
Fourier transform for magnetic resonance imaging (MRI), based on simple
mathematical models for the imaging systems. These methods are typically fast,
but have suboptimal properties such as poor resolution-noise trade-off for CT.
A second type is iterative reconstruction methods based on more complete models
for the imaging system physics and, where appropriate, models for the sensor
statistics. These iterative methods improved image quality by reducing noise
and artifacts. The FDA-approved methods among these have been based on
relatively simple regularization models. A third type of methods has been
designed to accommodate modified data acquisition methods, such as reduced
sampling in MRI and CT to reduce scan time or radiation dose. These methods
typically involve mathematical image models involving assumptions such as
sparsity or low-rank. A fourth type of methods replaces mathematically designed
models of signals and systems with data-driven or adaptive models inspired by
the field of machine learning. This paper focuses on the two most recent trends
in medical image reconstruction: methods based on sparsity or low-rank models,
and data-driven methods based on machine learning techniques.

本文介绍了医学图像重建方法中的四种类型，并聚焦于近期发展起来的基于稀疏性或低秩模型、以及基于机器学习技术的数据驱动方法。