The paper describes the open Russian medical language understanding benchmark
covering several task types (classification, question answering, natural
language inference, named entity recognition) on a number of novel text sets.
Given the sensitive nature of the data in healthcare, such a benchmark
partially closes the problem of Russian medical dataset absence. We prepare the
unified format labeling, data split, and evaluation metrics for new tasks. The
remaining tasks are from existing datasets with a few modifications. A
single-number metric expresses a model's ability to cope with the benchmark.
Moreover, we implement several baseline models, from simple ones to neural
networks with transformer architecture, and release the code. Expectedly, the
more advanced models yield better performance, but even a simple model is
enough for a decent result in some tasks. Furthermore, for all tasks, we
provide a human evaluation. Interestingly the models outperform humans in the
large-scale classification tasks. However, the advantage of natural
intelligence remains in the tasks requiring more knowledge and reasoning.

该论文描述了一个开放的俄罗斯医学语言理解基准，包括分类、问答、自然语言推理、命名实体识别等多种任务类型，以及新的文本集合，提供统一格式标注、数据分割和评估指标，还实现了多个基准模型，从简单模型到具有变形金刚架构的神经网络模型，预计高级模型表现更好，但即使是简单模型在某些任务上也可以得到不错的结果。