This paper introduces AnomaLLMy, a novel technique for the automatic
detection of anomalous tokens in black-box Large Language Models (LLMs) with
API-only access. Utilizing low-confidence single-token predictions as a
cost-effective indicator, AnomaLLMy identifies irregularities in model
behavior, addressing the issue of anomalous tokens degrading the quality and
reliability of models. Validated on the cl100k_base dataset, the token set of
GPT-4, AnomaLLMy detected 413 major and 65 minor anomalies, demonstrating the
method's efficiency with just \$24.39 spent in API credits. The insights from
this research are expected to be beneficial for enhancing the robustness of and
accuracy of LLMs, particularly in the development and assessment of tokenizers.

AnomaLLMy 是一种新颖的技术，用于通过 API 访问自动检测黑盒大型语言模型中的异常标记。通过利用低置信度的单词预测作为一种成本效益的指标，AnomaLLMy 识别模型行为中的异常，解决异常标记降低模型质量和可靠性的问题，通过在 cl100k_base 数据集上验证，AnomaLLMy 检测到了 413 个主要异常和 65 个次要异常，仅花费了 24.39 美元的 API 积分。本研究的发现有望增强大型语言模型的鲁棒性和准确性，特别是在分词器开发和评估方面。