Large Language Models(LLMs) have shown exceptional abilities, yet training
these models can be quite challenging. There is a strong dependence on the
quality of data and finding the best instruction tuning set. Further, the
inherent limitations in training methods create substantial difficulties to
train relatively smaller models with 7B and 13B parameters. In our research, we
suggest an improved training method for these models by utilising knowledge
from larger models, such as a mixture of experts (8x7B) architectures. The
scale of these larger models allows them to capture a wide range of variations
from data alone, making them effective teachers for smaller models. Moreover,
we implement a novel post-training domain alignment phase that employs
domain-specific expert models to boost domain-specific knowledge during
training while preserving the model's ability to generalise. Fine-tuning
Mistral 7B and 2x7B with our method surpasses the performance of
state-of-the-art language models with more than 7B and 13B parameters:
achieving up to $7.9$ in MT-Bench and $93.04\%$ on AlpacaEval.

我们提出了一种改进的训练方法，使用更大的模型的知识来提高大型语言模型的训练效果，同时利用领域专家模型进行领域对齐，通过这种方法，在性能上超过了具有更大参数数量的现有语言模型。

一位教师胜过百万句指令

A Teacher Is Worth A Million Instructions

Large language models, such as GPT-4 and Med-PaLM, have shown impressive
performance on clinical tasks; however, they require access to compute, are
closed-source, and cannot be deployed on device. Mid-size models such as
BioGPT-large, BioMedLM, LLaMA 2, and Mistral 7B avoid these drawbacks, but
their capacity for clinical tasks has been understudied. To help assess their
potential for clinical use and help researchers decide which model they should
use, we compare their performance on two clinical question-answering (QA)
tasks: MedQA and consumer query answering. We find that Mistral 7B is the best
performing model, winning on all benchmarks and outperforming models trained
specifically for the biomedical domain. While Mistral 7B's MedQA score of 63.0%
approaches the original Med-PaLM, and it often can produce plausible responses
to consumer health queries, room for improvement still exists. This study
provides the first head-to-head assessment of open source mid-sized models on
clinical tasks.

比较了大型语言模型和中型开源模型在临床任务上的表现，发现 Mistral 7B 是表现最佳的模型，并且在临床问题回答任务和消费者查询回答任务上优于专门为生物医学领域训练的模型。

评估中型语言模型在临床问答中的潜力

Assessing The Potential Of Mid-Sized Language Models For Clinical QA

This paper presents the outcomes of fine-tuning Mistral 7B, a general-purpose
large language model (LLM), for adaptive machine translation (MT). The
fine-tuning process involves utilising a combination of zero-shot and one-shot
translation prompts within the medical domain. The primary objective is to
enhance real-time adaptive MT capabilities of Mistral 7B, enabling it to adapt
translations to the required domain at inference time. The results,
particularly for Spanish-to-English MT, showcase the efficacy of the fine-tuned
model, demonstrating quality improvements in both zero-shot and one-shot
translation scenarios, surpassing Mistral 7B's baseline performance. Notably,
the fine-tuned Mistral outperforms ChatGPT "gpt-3.5-turbo" in zero-shot
translation while achieving comparable one-shot translation quality. Moreover,
the zero-shot translation of the fine-tuned Mistral matches NLLB 3.3B's
performance, and its one-shot translation quality surpasses that of NLLB 3.3B.
These findings emphasise the significance of fine-tuning efficient LLMs like
Mistral 7B to yield high-quality zero-shot translations comparable to
task-oriented models like NLLB 3.3B. Additionally, the adaptive gains achieved
in one-shot translation are comparable to those of commercial LLMs such as
ChatGPT. Our experiments demonstrate that, with a relatively small dataset of
20,000 segments that incorporate a mix of zero-shot and one-shot prompts,
fine-tuning significantly enhances Mistral's in-context learning ability,
especially for real-time adaptive MT.

利用零射击和单射击翻译提示在医学领域内进行了 Mistral 7B 大型通用语言模型 (LLM) 的微调，以提高其实时自适应机器翻译能力，结果显示其对西班牙语到英语机器翻译的质量改进超过了基准性能，并且其零射击翻译与 NLLB 3.3B 的性能相匹配，单射击翻译质量超过了 NLLB 3.3B 的性能，明确了微调高效 LLMs 像 Mistral 7B 的重要性，以产生与任务导向模型相当的高质量零射击翻译。

大规模语言模型微调的自适应机器翻译

Fine-tuning Large Language Models for Adaptive Machine Translation

We introduce Mistral 7B v0.1, a 7-billion-parameter language model engineered
for superior performance and efficiency. Mistral 7B outperforms Llama 2 13B
across all evaluated benchmarks, and Llama 1 34B in reasoning, mathematics, and
code generation. Our model leverages grouped-query attention (GQA) for faster
inference, coupled with sliding window attention (SWA) to effectively handle
sequences of arbitrary length with a reduced inference cost. We also provide a
model fine-tuned to follow instructions, Mistral 7B -- Instruct, that surpasses
the Llama 2 13B -- Chat model both on human and automated benchmarks. Our
models are released under the Apache 2.0 license.

Mistral 7B v0.1 是一个拥有 70 亿参数的语言模型，通过使用分组查询注意力（GQA）和滑动窗口注意力（SWA）提高了推理效率，并且还提供了一个经过调整的模型 Mistral 7B -- Instruct，在人类和自动化评测中都超过了 Llama 2 13B -- Chat 模型。