We show that marginals of subchains of length $t$ of any finitely correlated
translation invariant state on a chain can be learned, in trace distance, with
$O(t^2)$ copies -- with an explicit dependence on local dimension, memory
dimension and spectral properties of a certain map constructed from the state
-- and computational complexity polynomial in $t$. The algorithm requires only
the estimation of a marginal of a controlled size, in the worst case bounded by
a multiple of the minimum bond dimension, from which it reconstructs a
translation invariant matrix product operator. In the analysis, a central role
is played by the theory of operator systems. A refined error bound can be
proven for $C^*$-finitely correlated states, which have an operational
interpretation in terms of sequential quantum channels applied to the memory
system. We can also obtain an analogous error bound for a class of matrix
product density operators reconstructible by local marginals. In this case, a
linear number of marginals must be estimated, obtaining a sample complexity of
$\tilde{O}(t^3)$. The learning algorithm also works for states that are only
close to a finitely correlated state, with the potential of providing
competitive algorithms for other interesting families of states.

可以用大小受控制的边际重构出一种矩阵乘积算子，并证明了针对 C*- 有限相关的状态可以获得一个更精确的误差界，此外还可以应用于其他有趣状态的竞争性算法。

学习有限相关状态：谱重构的稳定性

Learning finitely correlated states: stability of the spectral  reconstruction

Recently, Mixture-of-Experts (short as MoE) architecture has achieved
remarkable success in increasing the model capacity of large-scale language
models. However, MoE requires incorporating significantly more parameters than
the base model being extended. In this paper, we propose building a
parameter-efficient MoE architecture by sharing information among experts. We
adopt the matrix product operator (MPO, a tensor decomposition from quantum
many-body physics) to reconstruct the parameter matrix in the expert layer and
increase model capacity for pre-trained language models by sharing parameters
of the central tensor (containing the core information) among different experts
while enabling the specificity through the auxiliary tensors (complementing the
central tensor) of different experts. To address the unbalanced optimization
issue, we further design the gradient mask strategy for the MPO-based MoE
architecture. Extensive experiments based on T5 and GPT-2 show improved
performance and efficiency of the pre-trained language model (27.2x reduction
in total parameters for the superior model performance, compared with the
Switch Transformers). Our code is publicly available at
this https URL.

本文提出了一种参数高效的混合专家架构，通过在专家层中共享参数矩阵中心张量的信息，并通过辅助张量增加各个专家的特异性，从而通过矩阵乘积算子来实现受量子多体物理学影响的张量分解，以解决混合专家架构中存在的参数膨胀问题，实验结果表明新方法具有更好的性能和效率。

预训练语言模型的参数高效专家混合架构

Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models

This paper presents a novel pre-trained language models (PLM) compression
approach based on the matrix product operator (short as MPO) from quantum
many-body physics. It can decompose an original matrix into central tensors
(containing the core information) and auxiliary tensors (with only a small
proportion of parameters). With the decomposed MPO structure, we propose a
novel fine-tuning strategy by only updating the parameters from the auxiliary
tensors, and design an optimization algorithm for MPO-based approximation over
stacked network architectures. Our approach can be applied to the original or
the compressed PLMs in a general way, which derives a lighter network and
significantly reduces the parameters to be fine-tuned. Extensive experiments
have demonstrated the effectiveness of the proposed approach in model
compression, especially the reduction in finetuning parameters (91% reduction
on average).

提出了一种基于量子多体物理中的矩阵积算符（简称 MPO）的新型预训练语言模型压缩方法，其可以将原始的矩阵分解成中心张量和辅助张量，通过仅更新辅助张量的参数进行微调并为 MPO 结构设计优化算法，可应用于原始或压缩的语言模型以实现更轻量级的网络和显著减少微调参数。