In the rapidly evolving field of artificial intelligence, large language
models (LLMs) have demonstrated significant capabilities across numerous
applications. However, the performance of these models in languages with fewer
resources, such as Swedish, remains under-explored. This study introduces a
comprehensive human benchmark to assess the efficacy of prominent LLMs in
understanding and generating Swedish language texts using forced choice
ranking. We employ a modified version of the ChatbotArena benchmark,
incorporating human feedback to evaluate eleven different models, including
GPT-4, GPT-3.5, various Claude and Llama models, and bespoke models like
Dolphin-2.9-llama3b-8b-flashback and BeagleCatMunin. These models were chosen
based on their performance on LMSYS chatbot arena and the Scandeval benchmarks.
We release the chatbotarena.se benchmark as a tool to improve our understanding
of language model performance in Swedish with the hopes that it will be widely
used. We aim to create a leaderboard once sufficient data has been collected
and analysed.

在人工智能领域，大型语言模型在多个应用中展示出显著的能力。然而，这些模型在资源较少的语言（如瑞典语）中的表现尚未深入研究。本研究引入了一种综合的人类基准，通过强制选择排序来评估主要的语言模型在理解和生成瑞典语文本方面的有效性。我们使用改进的 ChatbotArena 基准测试，结合人类反馈来评估包括 GPT-4、GPT-3.5、各种 Claude 和 Llama 模型以及定制模型（如 Dolphin-2.9-llama3b-8b-flashback 和 BeagleCatMunin）等十一种不同模型的性能。这些模型是基于它们在 LMSYS chatbot arena 和 Scandeval 基准测试中的表现选择的。我们发布 chatbotarena.se 基准测试作为一种工具，以改善我们对瑞典语语言模型性能的理解，并希望它能被广泛使用。我们的目标是在收集和分析足够的数据后创建一个排行榜。

通过人工反馈评估大型语言模型：建立瑞典基准

Evaluating Large Language Models with Human Feedback: Establishing a  Swedish Benchmark

This paper explains our work in developing new acoustic models for automated
speech recognition (ASR) at KBLab, the infrastructure for data-driven research
at the National Library of Sweden (KB). We evaluate different approaches for a
viable speech-to-text pipeline for audiovisual resources in Swedish, using the
wav2vec 2.0 architecture in combination with speech corpuses created from KB's
collections. These approaches include pretraining an acoustic model for Swedish
from the ground up, and fine-tuning existing monolingual and multilingual
models. The collections-based corpuses we use have been sampled from millions
of hours of speech, with a conscious attempt to balance regional dialects to
produce a more representative, and thus more democratic, model. The acoustic
model this enabled, "VoxRex", outperforms existing models for Swedish ASR. We
also evaluate combining this model with various pretrained language models,
which further enhanced performance. We conclude by highlighting the potential
of such technology for cultural heritage institutions with vast collections of
previously unlabelled audiovisual data. Our models are released for further
exploration and research here: this https URL.

本文通过使用 wav2vec 2.0 结构与来自瑞典国家图书馆 (KB) 收藏的语音语料库，对实现适用于瑞典语音频资源的语音文本管道的不同方法进行了评估并进行了改进。最终，作者提出的 VoxRex 声学模型表现出比现有瑞典语言 ASR 模型更好的性能，并强调了这样的技术在文化遗产机构中处理大量未标记音频视觉数据的潜力。

国家图书馆内的言语语料库与瑞典语言音模型

Hearing voices at the National Library -- a speech corpus and acoustic model for the Swedish language

Building open-domain conversational systems (or chatbots) that produce
convincing responses is a recognized challenge. Recent state-of-the-art (SoTA)
transformer-based models for the generation of natural language dialogue have
demonstrated impressive performance in simulating human-like, single-turn
conversations in English. This work investigates, by an empirical study, the
potential for transfer learning of such models to Swedish language. DialoGPT,
an English language pre-trained model, is adapted by training on three
different Swedish language conversational datasets obtained from publicly
available sources. Perplexity score (an automated intrinsic language model
metric) and surveys by human evaluation were used to assess the performances of
the fine-tuned models, with results that indicate that the capacity for
transfer learning can be exploited with considerable success. Human evaluators
asked to score the simulated dialogue judged over 57% of the chatbot responses
to be human-like for the model trained on the largest (Swedish) dataset. We
provide the demos and model checkpoints of our English and Swedish chatbots on
the HuggingFace platform for public use.

对最近的状态转化器自然语言生成对话模型进行实证研究，探讨了其在瑞典语言方面的迁移学习潜力，结果表明通过相应 fine-tuned，该模型具备可观的成功移植能力，并提供了英文与瑞典语聊天机器人的演示和模型检查点以供公众使用。