Collaborative filtering (CF) has been proven to be one of the most effective
techniques for recommendation. Among all CF approaches, SimpleX is the
state-of-the-art method that adopts a novel loss function and a proper number
of negative samples. However, there is no work that optimizes SimpleX on
multi-core CPUs, leading to limited performance. To this end, we perform an
in-depth profiling and analysis of existing SimpleX implementations and
identify their performance bottlenecks including (1) irregular memory accesses,
(2) unnecessary memory copies, and (3) redundant computations. To address these
issues, we propose an efficient CF training system (called HEAT) that fully
enables the multi-level caching and multi-threading capabilities of modern
CPUs. Specifically, the optimization of HEAT is threefold: (1) It tiles the
embedding matrix to increase data locality and reduce cache misses (thus reduce
read latency); (2) It optimizes stochastic gradient descent (SGD) with sampling
by parallelizing vector products instead of matrix-matrix multiplications, in
particular the similarity computation therein, to avoid memory copies for
matrix data preparation; and (3) It aggressively reuses intermediate results
from the forward phase in the backward phase to alleviate redundant
computation. Evaluation on five widely used datasets with both x86- and
ARM-architecture processors shows that HEAT achieves up to 65.3X speedup over
existing CPU solution and 4.8X speedup and 7.9X cost reduction in Cloud over
existing GPU solution with NVIDIA V100 GPU.

本文针对 Collaborative filtering 中的 state-of-the-art 方法 SimpleX 在 multi-core CPUs 表现欠佳的问题，提出了一种名为 HEAT 的高效 CF 训练系统，通过瓦片 embedding 矩阵、优化 SGD 方法和重复利用前向阶段结果的方式有效提升了计算性能，并在 5 个常用数据集上进行了评估。结果表明，在 x86 - 和 ARM - 系 CPU 上，HEAT 相对现有 CPU 解决方案提升了高达 65.3 倍的速度，在使用 NVIDIA V100 GPU 的云计算方案上，也取得了 4.8 倍的加速和 7.9 倍的成本降低。

HEAT: 面向 CPU 的协同过滤推荐高效易用的训练系统

HEAT: A Highly Efficient and Affordable Training System for  Collaborative Filtering Based Recommendation on CPUs

Transformers have attained superior performance in natural language
processing and computer vision. Their self-attention and feedforward layers are
overparameterized, limiting inference speed and energy efficiency. Tensor
decomposition is a promising technique to reduce parameter redundancy by
leveraging tensor algebraic properties to express the parameters in a
factorized form. Prior efforts used manual or heuristic factorization settings
without hardware-aware customization, resulting in poor hardware efficiencies
and large performance degradation.
In this work, we propose a hardware-aware tensor decomposition framework,
dubbed HEAT, that enables efficient exploration of the exponential space of
possible decompositions and automates the choice of tensorization shape and
decomposition rank with hardware-aware co-optimization. We jointly investigate
tensor contraction path optimizations and a fused Einsum mapping strategy to
bridge the gap between theoretical benefits and real hardware efficiency
improvement. Our two-stage knowledge distillation flow resolves the
trainability bottleneck and thus significantly boosts the final accuracy of
factorized Transformers. Overall, we experimentally show that our
hardware-aware factorized BERT variants reduce the energy-delay product by 5.7x
with less than 1.1% accuracy loss and achieve a better efficiency-accuracy
Pareto frontier than hand-tuned and heuristic baselines.

本文提出了 HEAT 硬件感知张量分解框架，通过硬件感知的协同优化，实现了对参数冗余的降维和的自动调节，能够达到更好的能效比和精度。