Modern deep neural networks (DNNs) often require high memory consumption and
large computational loads. In order to deploy DNN algorithms efficiently on
edge or mobile devices, a series of DNN compression algorithms have been
explored, including factorization methods. Factorization methods approximate
the weight matrix of a DNN layer with the multiplication of two or multiple
low-rank matrices. However, it is hard to measure the ranks of DNN layers
during the training process. Previous works mainly induce low-rank through
implicit approximations or via costly singular value decomposition (SVD)
process on every training step. The former approach usually induces a high
accuracy loss while the latter has a low efficiency. In this work, we propose
SVD training, the first method to explicitly achieve low-rank DNNs during
training without applying SVD on every step. SVD training first decomposes each
layer into the form of its full-rank SVD, then performs training directly on
the decomposed weights. We add orthogonality regularization to the singular
vectors, which ensure the valid form of SVD and avoid gradient
vanishing/exploding. Low-rank is encouraged by applying sparsity-inducing
regularizers on the singular values of each layer. Singular value pruning is
applied at the end to explicitly reach a low-rank model. We empirically show
that SVD training can significantly reduce the rank of DNN layers and achieve
higher reduction on computation load under the same accuracy, comparing to not
only previous factorization methods but also state-of-the-art filter pruning
methods.

本研究提出了一种名为 SVD training 的神经网络压缩方法，通过正交性正则化和奇异值剪枝等技术，可以在训练过程中显式地实现降低矩阵秩的目标，从而更有效地减少 DNN 算法在低性能设备上的计算负担。

通过奇异向量正交正则化和奇异值稀疏化学习低秩深度神经网络

Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality  Regularization and Singular Value Sparsification

Graphs, such as social networks, word co-occurrence networks, and
communication networks, occur naturally in various real-world applications.
Analyzing them yields insight into the structure of society, language, and
different patterns of communication. Many approaches have been proposed to
perform the analysis. Recently, methods which use the representation of graph
nodes in vector space have gained traction from the research community. In this
survey, we provide a comprehensive and structured analysis of various graph
embedding techniques proposed in the literature. We first introduce the
embedding task and its challenges such as scalability, choice of
dimensionality, and features to be preserved, and their possible solutions. We
then present three categories of approaches based on factorization methods,
random walks, and deep learning, with examples of representative algorithms in
each category and analysis of their performance on various tasks. We evaluate
these state-of-the-art methods on a few common datasets and compare their
performance against one another. Our analysis concludes by suggesting some
potential applications and future directions. We finally present the
open-source Python library we developed, named GEM (Graph Embedding Methods,
available at this https URL), which provides all presented
algorithms within a unified interface to foster and facilitate research on the
topic.

本文综述了图嵌入技术的各种方法，介绍了三个不同的分类方法的代表性算法，并分析了它们在不同任务上的表现。文章最终结论是提出了一些潜在的应用和未来方向，并介绍了一个名为 GEM 的 Python 库，其中提供了所有介绍的算法作为一个统一的接口，以促进和便利这个领域的研究。

图嵌入技术、应用及性能：一项调查

Graph Embedding Techniques, Applications, and Performance: A Survey

Context-aware recommendation algorithms focus on refining recommendations by
considering additional information, available to the system. This topic has
gained a lot of attention recently. Among others, several factorization methods
were proposed to solve the problem, although most of them assume explicit
feedback which strongly limits their real-world applicability. While these
algorithms apply various loss functions and optimization strategies, the
preference modeling under context is less explored due to the lack of tools
allowing for easy experimentation with various models. As context dimensions
are introduced beyond users and items, the space of possible preference models
and the importance of proper modeling largely increases.
In this paper we propose a General Factorization Framework (GFF), a single
flexible algorithm that takes the preference model as an input and computes
latent feature matrices for the input dimensions. GFF allows us to easily
experiment with various linear models on any context-aware recommendation task,
be it explicit or implicit feedback based. The scaling properties makes it
usable under real life circumstances as well.
We demonstrate the framework's potential by exploring various preference
models on a 4-dimensional context-aware problem with contexts that are
available for almost any real life datasets. We show in our experiments --
performed on five real life, implicit feedback datasets -- that proper
preference modelling significantly increases recommendation accuracy, and
previously unused models outperform the traditional ones. Novel models in GFF
also outperform state-of-the-art factorization algorithms.
We also extend the method to be fully compliant to the Multidimensional
Dataspace Model, one of the most extensive data models of context-enriched
data. Extended GFF allows the seamless incorporation of information into the
fac[truncated]

本文提出了一种通用的分解框架（GFF），通过使用此框架，作者在处理推荐任务时探索了各种偏好模型。实验结果表明，正确的偏好建模显著提高了推荐准确性，并且 GFF 中的新模型优于其他状态下的最先进的分解算法。