With the concept of teaching being introduced to the machine learning
community, a teacher model start using dynamic loss functions to teach the
training of a student model. The dynamic intends to set adaptive loss functions
to different phases of student model learning. In existing works, the teacher
model 1) merely determines the loss function based on the present states of the
student model, i.e., disregards the experience of the teacher; 2) only utilizes
the states of the student model, e.g., training iteration number and
loss/accuracy from training/validation sets, while ignoring the states of the
loss function. In this paper, we first formulate the loss adjustment as a
temporal task by designing a teacher model with memory units, and, therefore,
enables the student learning to be guided by the experience of the teacher
model. Then, with a dynamic loss network, we can additionally use the states of
the loss to assist the teacher learning in enhancing the interactions between
the teacher and the student model. Extensive experiments demonstrate our
approach can enhance student learning and improve the performance of various
deep models on real-world tasks, including classification, objective detection,
and semantic segmentation scenarios.

此研究论文介绍了使用动态损失函数来指导学生模型训练的教师模型，通过设计带有记忆单元的教师模型，利用教师的经验引导学生学习，并通过动态损失网络利用损失的状态来增强教师和学生模型之间的交互，实验结果证明该方法可以提高学生模型的学习性能和在实际任务中的表现。

L2T-DLN：学习使用动态损失网络进行教学

L2T-DLN: Learning to Teach with Dynamic Loss Network

Memory units have been widely used to enrich the capabilities of deep
networks on capturing long-term dependencies in reasoning and prediction tasks,
but little investigation exists on deep generative models (DGMs) which are good
at inferring high-level invariant representations from unlabeled data. This
paper presents a deep generative model with a possibly large external memory
and an attention mechanism to capture the local detail information that is
often lost in the bottom-up abstraction process in representation learning. By
adopting a smooth attention model, the whole network is trained end-to-end by
optimizing a variational bound of data likelihood via auto-encoding variational
Bayesian methods, where an asymmetric recognition network is learnt jointly to
infer high-level invariant representations. The asymmetric architecture can
reduce the competition between bottom-up invariant feature extraction and
top-down generation of instance details. Our experiments on several datasets
demonstrate that memory can significantly boost the performance of DGMs and
even achieve state-of-the-art results on various tasks, including density
estimation, image generation, and missing value imputation.

该研究提出一种深度生成模型，在表示学习中采用了注意机制和大型外部记忆来捕捉局部详细信息，具有变分自编码贝叶斯方法的数据似然度上限，以及学习高级不变表示的非对称识别网络。实验结果表明，记忆单元可显着提高 DGMs 的性能，并在各种任务中达到最先进的结果。

记忆式生成学习

Learning to Generate with Memory

We study an indexing architecture to store and search in a database of
high-dimensional vectors from the perspective of statistical signal processing
and decision theory. This architecture is composed of several memory units,
each of which summarizes a fraction of the database by a single representative
vector. The potential similarity of the query to one of the vectors stored in
the memory unit is gauged by a simple correlation with the memory unit's
representative vector. This representative optimizes the test of the following
hypothesis: the query is independent from any vector in the memory unit vs. the
query is a simple perturbation of one of the stored vectors.
Compared to exhaustive search, our approach finds the most similar database
vectors significantly faster without a noticeable reduction in search quality.
Interestingly, the reduction of complexity is provably better in
high-dimensional spaces. We empirically demonstrate its practical interest in a
large-scale image search scenario with off-the-shelf state-of-the-art
descriptors.

研究一个索引架构，从统计信号处理和决策理论角度，存储和搜索高维向量数据库。该架构由多个内存单元组成，每个内存单元通过单个代表向量总结一部分数据库。和穷举搜索相比，我们的方法能够更快地找到相似的数据库向量，而不会显著降低搜索质量。