Model checkpoints are critical Deep Learning (DL) artifacts that enable fault
tolerance for training and downstream applications, such as inference. However,
writing checkpoints to persistent storage, and other I/O aspects of DL
training, are mostly ignored by compute-focused optimization efforts for faster
training of rapidly growing models and datasets. Towards addressing this
imbalance, we propose FastPersist to accelerate checkpoint creation in DL
training. FastPersist combines three novel techniques: (i) NVMe optimizations
for faster checkpoint writes to SSDs, (ii) efficient write parallelism using
the available SSDs in training environments, and (iii) overlapping
checkpointing with independent training computations. Our evaluation using real
world dense and sparse DL models shows that FastPersist creates checkpoints in
persistent storage up to 116x faster than baseline, and enables per-iteration
checkpointing with negligible overhead.

FastPersist 通过结合 NVMe 的优化、有效的写并行化以及与独立的训练计算重叠的检查点创建方式，实现了在持久存储中创建检查点的加速，相比基线方法快速 116 倍，并实现了每次迭代检查点的无可忽略开销。

FastPersist：加速深度学习中的模型检查点

FastPersist: Accelerating Model Checkpointing in Deep Learning

In this paper, we introduce summarization MevakerSumm and conclusion
extraction MevakerConc datasets for the Hebrew language based on the State
Comptroller and Ombudsman of Israel reports, along with two auxiliary datasets.
We accompany these datasets with models for conclusion extraction (HeConE,
HeConEspc) and conclusion allocation (HeCross). All of the code, datasets, and
model checkpoints used in this work are publicly available.

该论文介绍了基于以色列国家审计长与公民监察员报告的 MeVakerSumm 和 MeVakerConc 数据集，以及两个辅助数据集，同时提供了用于结论提取（HeConE，HeConEspc）和结论分配（HeCross）的模型。本论文中使用的所有代码、数据集和模型检查点均已公开。

Mevaker: 希伯来语的结论提取和资源分配

Mevaker: Conclusion Extraction and Allocation Resources for the Hebrew  Language

While recent preference alignment algorithms for language models have
demonstrated promising results, supervised fine-tuning (SFT) remains imperative
for achieving successful convergence. In this paper, we study the crucial role
of SFT within the context of preference alignment, emphasizing that a minor
penalty for the disfavored generation style is sufficient for
preference-aligned SFT. Building on this foundation, we introduce a
straightforward and innovative reference model-free monolithic odds ratio
preference optimization algorithm, ORPO, eliminating the necessity for an
additional preference alignment phase. We demonstrate, both empirically and
theoretically, that the odds ratio is a sensible choice for contrasting favored
and disfavored styles during SFT across the diverse sizes from 125M to 7B.
Specifically, fine-tuning Phi-2 (2.7B), Llama-2 (7B), and Mistral (7B) with
ORPO on the UltraFeedback alone surpasses the performance of state-of-the-art
language models with more than 7B and 13B parameters: achieving up to 12.20% on
$\text{AlpacaEval}_{2.0}$ and 7.32 in MT-Bench, as shown in Figures 1 and 12.
We release code and model checkpoints for Mistral-ORPO-$\alpha$ (7B) and
Mistral-ORPO-$\beta$ (7B).

优先对齐算法、受监督的微调 (SFT)、单体化比值偏好优化算法 (ORPO)、最新的语言模型和模型检查点

无参考单块偏好优化与胜负比率

Reference-free Monolithic Preference Optimization with Odds Ratio

Massively multilingual language models have displayed strong performance in
zero-shot (ZS-XLT) and few-shot (FS-XLT) cross-lingual transfer setups, where
models fine-tuned on task data in a source language are transferred without any
or with only a few annotated instances to the target language(s). However,
current work typically overestimates model performance as fine-tuned models are
frequently evaluated at model checkpoints that generalize best to validation
instances in the target languages. This effectively violates the main
assumptions of "true" ZS-XLT and FS-XLT. Such XLT setups require robust methods
that do not depend on labeled target language data for validation and model
selection. In this work, aiming to improve the robustness of "true" ZS-XLT and
FS-XLT, we propose a simple and effective method that averages different
checkpoints (i.e., model snapshots) during task fine-tuning. We conduct
exhaustive ZS-XLT and FS-XLT experiments across higher-level semantic tasks
(NLI, extractive QA) and lower-level token classification tasks (NER, POS). The
results indicate that averaging model checkpoints yields systematic and
consistent performance gains across diverse target languages in all tasks.
Importantly, it simultaneously substantially desensitizes XLT to varying
hyperparameter choices in the absence of target language validation. We also
show that checkpoint averaging benefits performance when further combined with
run averaging (i.e., averaging the parameters of models fine-tuned over
independent runs).

本文探讨了在跨语种任务上，fine-tuning 的模型在模型点上的性能评估并不符合 zero-shot 和 few-shot 学习的前提；并提出了一种更为稳健的模型检查点平均方法，特别是在没有标注的目标语言数据的情况下，该方法在多个任务和语种下表现出可靠的性能。