MeetEval is an open-source toolkit to evaluate all kinds of meeting transcription systems. It provides a unified interface for the computation of commonly used Word Error Rates (WERs), specifically cpWER, ORC WER and MIMO WER along other WER definitions. We extend the cpWER computation by a temporal constraint to ensure that only words are identified as correct when the temporal alignment is plausible. This leads to a better quality of the matching of the hypothesis string to the reference string that more closely resembles the actual transcription quality, and a system is penalized if it provides poor time annotations. Since word-level timing information is often not available, we present a way to approximate exact word-level timings from segment-level timings (e.g., a sentence) and show that the approximation leads to a similar WER as a matching with exact word-level annotations. At the same time, the time constraint leads to a speedup of the matching algorithm, which outweighs the additional overhead caused by processing the time stamps.

MeetEval是一个开源工具包，用于评估各种会议转录系统。它提供了一个统一的界面，用于计算常用的词错误率（WER），包括cpWER、ORC WER和MIMO WER等。我们通过时间约束扩展了cpWER的计算，以确保只有当时间对齐是合理的时候，才将单词视为正确。这从更接近实际转录质量的参考字符串匹配上，提供了更好的匹配质量，如果系统提供了较差的时间注释，将会受到惩罚。由于单词级别的时间信息通常不可用，我们提出了一种从片段级别时间（例如句子）近似确定确切单词级别时间的方法，并展示了近似法与具有准确单词级注释的匹配导致类似的词错误率。同时，时间约束导致匹配算法的加速，这抵消了处理时间戳引起的额外开销。

MeetEval：会议转录系统的词错误率计算工具包