Large language models (LLMs) have become crucial for many generative
downstream tasks, leading to an inevitable trend and significant challenge to
deploy them efficiently on resource-constrained devices. Structured pruning is
a widely used method to address this challenge. However, when dealing with the
complex structure of the multiple decoder layers, general methods often employ
common estimation approaches for pruning. These approaches lead to a decline in
accuracy for specific downstream tasks. In this paper, we introduce a simple
yet efficient method that adaptively models the importance of each
substructure. Meanwhile, it can adaptively fuse coarse-grained and finegrained
estimations based on the results from complex and multilayer structures. All
aspects of our design seamlessly integrate into the endto-end pruning
framework. Our experimental results, compared with state-of-the-art methods on
mainstream datasets, demonstrate average accuracy improvements of 1.1%, 1.02%,
2.0%, and 1.2% for LLaMa-7B,Vicuna-7B, Baichuan-7B, and Bloom-7b1,
respectively.

用于资源受限设备的大型语言模型结构剪枝方法，在多层结构的基础上，通过自适应建模和融合估计结果实现了对每个子结构的重要性的自适应调整，实验结果表明在主流数据集上相比最先进的方法，平均准确率提高了 1.1％，1.02％，2.0％和 1.2％。

大规模语言模型的高效剪枝与自适应推理融合

Efficient Pruning of Large Language Model with Adaptive Estimation  Fusion

Spatio-temporal point process (STPP) is a stochastic collection of events
accompanied with time and space. Due to computational complexities, existing
solutions for STPPs compromise with conditional independence between time and
space, which consider the temporal and spatial distributions separately. The
failure to model the joint distribution leads to limited capacities in
characterizing the spatio-temporal entangled interactions given past events. In
this work, we propose a novel parameterization framework for STPPs, which
leverages diffusion models to learn complex spatio-temporal joint
distributions. We decompose the learning of the target joint distribution into
multiple steps, where each step can be faithfully described by a Gaussian
distribution. To enhance the learning of each step, an elaborated
spatio-temporal co-attention module is proposed to capture the interdependence
between the event time and space adaptively. For the first time, we break the
restrictions on spatio-temporal dependencies in existing solutions, and enable
a flexible and accurate modeling paradigm for STPPs. Extensive experiments from
a wide range of fields, such as epidemiology, seismology, crime, and urban
mobility, demonstrate that our framework outperforms the state-of-the-art
baselines remarkably, with an average improvement of over 50%. Further in-depth
analyses validate its ability to capture spatio-temporal interactions, which
can learn adaptively for different scenarios. The datasets and source code are
available online:
this https URL

该研究提出了一种新的基于扩散模型的参数化框架，用于学习复杂的时空联合分布，它打破了现有解决方案对时空依赖关系的限制，并实现了灵活准确的预测。

时空扩散点过程

Spatio-temporal Diffusion Point Processes

Many real-world problems, including multi-speaker text-to-speech synthesis,
can greatly benefit from the ability to meta-learn large models with only a few
task-specific components. Updating only these task-specific modules then allows
the model to be adapted to low-data tasks for as many steps as necessary
without risking overfitting. Unfortunately, existing meta-learning methods
either do not scale to long adaptation or else rely on handcrafted
task-specific architectures. Here, we propose a meta-learning approach that
obviates the need for this often sub-optimal hand-selection. In particular, we
develop general techniques based on Bayesian shrinkage to automatically
discover and learn both task-specific and general reusable modules.
Empirically, we demonstrate that our method discovers a small set of meaningful
task-specific modules and outperforms existing meta-learning approaches in
domains like few-shot text-to-speech that have little task data and long
adaptation horizons. We also show that existing meta-learning methods including
MAML, iMAML, and Reptile emerge as special cases of our method.

提出了一种能够自动发现并学习任务特定和通用可重用模块的元学习方法，以实现针对低数据任务的长时间自适应模型，适用于少样本 TTS 等通常存在少量数据和长时间自适应问题的领域，并在实验证明其表现优于现有元学习方法，包括 MAML，iMAML 和 Reptile。