Online Continual learning is a challenging learning scenario where the model
must learn from a non-stationary stream of data where each sample is seen only
once. The main challenge is to incrementally learn while avoiding catastrophic
forgetting, namely the problem of forgetting previously acquired knowledge
while learning from new data. A popular solution in these scenario is to use a
small memory to retain old data and rehearse them over time. Unfortunately, due
to the limited memory size, the quality of the memory will deteriorate over
time. In this paper we propose OLCGM, a novel replay-based continual learning
strategy that uses knowledge condensation techniques to continuously compress
the memory and achieve a better use of its limited size. The sample
condensation step compresses old samples, instead of removing them like other
replay strategies. As a result, the experiments show that, whenever the memory
budget is limited compared to the complexity of the data, OLCGM improves the
final accuracy compared to state-of-the-art replay strategies.

本文提出了一种新的重放式连续学习策略 OLCGM，使用知识压缩技术持续压缩记忆并更好地利用其有限的内存大小，取得了比现有重放策略更高的最终准确性。

在线连续学习中的样本压缩

Sample Condensation in Online Continual Learning

Attention-based models have demonstrated remarkable success in various
natural language understanding tasks. However, efficient execution remains a
challenge for these models which are memory-bound due to their massive number
of parameters. We present GOBO, a model quantization technique that compresses
the vast majority (typically 99.9%) of the 32-bit floating-point parameters of
state-of-the-art BERT models and their variants to 3 bits while maintaining
their accuracy. Unlike other quantization methods, GOBO does not require
fine-tuning nor retraining to compensate for the quantization error. We present
two practical hardware applications of GOBO. In the first GOBO reduces memory
storage and traffic and as a result inference latency and energy consumption.
This GOBO memory compression mechanism is plug-in compatible with many
architectures; we demonstrate it with the TPU, Eyeriss, and an architecture
using Tensor Cores-like units. Second, we present a co-designed hardware
architecture that also reduces computation. Uniquely, the GOBO architecture
maintains most of the weights in 3b even during computation, a property that:
(1) makes the processing elements area efficient, allowing us to pack more
compute power per unit area, (2) replaces most multiply-accumulations with
additions, and (3) reduces the off-chip traffic by amplifying on-chip memory
capacity.

GOBO 是一种模型量化技术，将当前 BERT 模型的大部分参数从 32 位浮点数压缩到 3 位，无需调整或重新训练即可同时保持其准确性，在硬件上可用于内存压缩和流量减少.

GOBO: 基于量化的注意力自然语言处理模型，实现低延迟和高能效推断

GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy  Efficient Inference

An ideal cognitively-inspired memory system would compress and organize
incoming items. The Kanerva Machine (Wu et al, 2018) is a Bayesian model that
naturally implements online memory compression. However, the organization of
the Kanerva Machine is limited by its use of a single Gaussian random matrix
for storage. Here we introduce the Product Kanerva Machine, which dynamically
combines many smaller Kanerva Machines. Its hierarchical structure provides a
principled way to abstract invariant features and gives scaling and capacity
advantages over single Kanerva Machines. We show that it can exhibit
unsupervised clustering, find sparse and combinatorial allocation patterns, and
discover spatial tunings that approximately factorize simple images by object.

介绍了 Product Kanerva Machine，它是一种具有分层结构的贝叶斯模型，通过动态组合许多较小的 Kanerva Machines，实现了大规模存储和压缩。它展示了成簇、罕见和组合分配模式，并发现了近似于通过对象对简单图像进行因子分解的空间调整。

产品 Kanerva 机：分解贝叶斯记忆

Product Kanerva Machines: Factorized Bayesian Memory

We describe a large vocabulary speech recognition system that is accurate,
has low latency, and yet has a small enough memory and computational footprint
to run faster than real-time on a Nexus 5 Android smartphone. We employ a
quantized Long Short-Term Memory (LSTM) acoustic model trained with
connectionist temporal classification (CTC) to directly predict phoneme
targets, and further reduce its memory footprint using an SVD-based compression
scheme. Additionally, we minimize our memory footprint by using a single
language model for both dictation and voice command domains, constructed using
Bayesian interpolation. Finally, in order to properly handle device-specific
information, such as proper names and other context-dependent information, we
inject vocabulary items into the decoder graph and bias the language model
on-the-fly. Our system achieves 13.5% word error rate on an open-ended
dictation task, running with a median speed that is seven times faster than
real-time.

本文介绍了一种大词汇量语音识别系统，其特点是准确、延迟低，同时其内存和计算资源占用不大，可以在 Nexus 5 Android 智能手机上以快于实时的速度运行。使用一种量化的 LSTM 音频模型和 CTC 训练直接预测音素目标，进一步使用基于 SVD 的压缩方案进一步减小内存占用，同时利用贝叶斯插值构建单一的语言模型，在植入词汇项进入解码器图表并实时更改语言模型偏差的情况下正确执行设备特定的信息。其最终取得的效果是在开放式口述任务中 13.5% 的单词错误率，而以运行速度优于实时的为媒介获得更优秀的结果。