The rapid expansion of the open-source language model landscape presents an
opportunity to merge the competencies of these model checkpoints by combining
their parameters. Advances in transfer learning, the process of fine-tuning
pre-trained models for specific tasks, has resulted in the development of vast
amounts of task-specific models, typically specialized in individual tasks and
unable to utilize each other's strengths. Model merging facilitates the
creation of multitask models without the need for additional training, offering
a promising avenue for enhancing model performance and versatility. By
preserving the intrinsic capabilities of the original models, model merging
addresses complex challenges in AI - including the difficulties of catastrophic
forgetting and multi-task learning. To support this expanding area of research,
we introduce MergeKit, a comprehensive, open-source library designed to
facilitate the application of model merging strategies. MergeKit offers an
extensible framework to efficiently merge models on any hardware, providing
utility to researchers and practitioners. To date, thousands of models have
been merged by the open-source community, leading to the creation of some of
the worlds most powerful open-source model checkpoints, as assessed by the Open
LLM Leaderboard. The library is accessible at
this https URL

采用开源语言模型、迁移学习和模型合并技术，通过创建多任务模型提升性能和应用领域的研究。为了支持这一领域的发展，推出了名为 MergeKit 的开源库，该库提供了一个可扩展的框架，便于在任何硬件上高效合并模型。

Arcee 的合并工具包：一个用于合并大型语言模型的工具包

Arcee's MergeKit: A Toolkit for Merging Large Language Models

Modern automatic translation systems aim at place the human at the center by
providing contextual support and knowledge. In this context, a critical task is
enriching the output with information regarding the mentioned entities, which
is currently achieved processing the generated translation with named entity
recognition (NER) and entity linking systems. In light of the recent promising
results shown by direct speech translation (ST) models and the known weaknesses
of cascades (error propagation and additional latency), in this paper we
propose multitask models that jointly perform ST and NER, and compare them with
a cascade baseline. The experimental results show that our models significantly
outperform the cascade on the NER task (by 0.4-1.0 F1), without degradation in
terms of translation quality, and with the same computational efficiency of a
plain direct ST model.

本篇论文提出了多任务模型将直接语音翻译和命名实体识别相结合，并与传统串行方法进行了对比，在不降低翻译质量的情况下，在命名实体识别任务上显著提高了性能。

联合语音翻译和命名实体识别

Joint Speech Translation and Named Entity Recognition

We present ATOMIC, an atlas of everyday commonsense reasoning, organized
through 877k textual descriptions of inferential knowledge. Compared to
existing resources that center around taxonomic knowledge, ATOMIC focuses on
inferential knowledge organized as typed if-then relations with variables
(e.g., "if X pays Y a compliment, then Y will likely return the compliment").
We propose nine if-then relation types to distinguish causes vs. effects,
agents vs. themes, voluntary vs. involuntary events, and actions vs. mental
states. By generatively training on the rich inferential knowledge described in
ATOMIC, we show that neural models can acquire simple commonsense capabilities
and reason about previously unseen events. Experimental results demonstrate
that multitask models that incorporate the hierarchical structure of if-then
relation types lead to more accurate inference compared to models trained in
isolation, as measured by both automatic and human evaluation.

ATOMIC 是一种以常识推理为组织形式的图表，其中包括了 877k 个基于变量的 if-then 关系类型，使用这种类型，神经模型可以获得简单的常识能力并进行推理，证实了将多任务模型引入到 if-then 关系结构中可以得到比独立训练的模型更准确的推论。

ATOMIC: 机器常识图谱用于 If-Then 推理

ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning

We explore multitask models for neural translation of speech, augmenting them
in order to reflect two intuitive notions. First, we introduce a model where
the second task decoder receives information from the decoder of the first
task, since higher-level intermediate representations should provide useful
information. Second, we apply regularization that encourages transitivity and
invertibility. We show that the application of these notions on jointly trained
models improves performance on the tasks of low-resource speech transcription
and translation. It also leads to better performance when using attention
information for word discovery over unsegmented input.

本文探讨了用于神经语音翻译的多任务模型，并对其进行增强，以反映两个直观概念。精准的中间表示和神经关联的模式，参与模型的训练可以提高低资源语音转录和翻译任务的性能。