This paper will present textual corpora for Serbian (and Serbo-Croatian),
usable for the training of large language models and publicly available at one
of the several notable online repositories. Each corpus will be classified
using multiple methods and its characteristics will be detailed. Additionally,
the paper will introduce three new corpora: a new umbrella web corpus of
Serbo-Croatian, a new high-quality corpus based on the doctoral dissertations
stored within National Repository of Doctoral Dissertations from all
Universities in Serbia, and a parallel corpus of abstract translation from the
same source. The uniqueness of both old and new corpora will be accessed via
frequency-based stylometric methods, and the results will be briefly discussed.

该文章介绍了用于大型语言模型训练的塞尔维亚语（和塞尔维亚 - 克罗地亚语）的文本语料库，并公开存储在多个知名在线资源库之一。对每个语料库使用多种方法进行分类，并详细说明其特点。此外，文章还介绍了三个新的语料库：一个新的塞尔维亚 - 克罗地亚语综合网络语料库，一个基于塞尔维亚所有大学国家博士论文库中的博士学位论文的高质量语料库，以及同一来源的摘要翻译平行语料库。将通过基于频率的文体测量方法评估新旧语料库的独特性，并对结果进行简要讨论。

塞尔维亚语言建模的新文本语料库

New Textual Corpora for Serbian Language Modeling

Noticing the urgent need to provide tools for fast and user-friendly
qualitative analysis of large-scale textual corpora of the modern NLP, we
propose to turn to the mature and well-tested methods from the domain of
Information Retrieval (IR) - a research field with a long history of tackling
TB-scale document collections. We discuss how Pyserini - a widely used toolkit
for reproducible IR research can be integrated with the Hugging Face ecosystem
of open-source AI libraries and artifacts. We leverage the existing
functionalities of both platforms while proposing novel features further
facilitating their integration. Our goal is to give NLP researchers tools that
will allow them to develop retrieval-based instrumentation for their data
analytics needs with ease and agility. We include a Jupyter Notebook-based walk
through the core interoperability features, available on GitHub at
this https URL We then demonstrate how the ideas we
present can be operationalized to create a powerful tool for qualitative data
analysis in NLP. We present GAIA Search - a search engine built following
previously laid out principles, giving access to four popular large-scale text
collections. GAIA serves a dual purpose of illustrating the potential of
methodologies we discuss but also as a standalone qualitative analysis tool
that can be leveraged by NLP researchers aiming to understand datasets prior to
using them in training. GAIA is hosted live on Hugging Face Spaces -
this https URL

本篇论文提出使用信息检索领域的成熟和经过多次测试的方法来对现代 NLP 的大规模文本语料库进行快速和用户友好的质性分析，提出了 GAIA Search 作为一个在 NLP 数据质量分析中既可以作为原型工具又可以作为独立的定性分析工具的搜索引擎示例。