The advancement of large language models has significantly improved natural
language processing. However, challenges such as jailbreaks (prompt injections
that cause an LLM to follow instructions contrary to its intended use),
hallucinations (generating incorrect or misleading information), and
comprehension errors remain prevalent. In this report, we present a comparative
analysis of the performance of fifteen distinct models, with each model
undergoing a standardized test comprising 38 queries across three key metrics:
jailbreaks, hallucinations, and comprehension errors. The models are assessed
based on the total occurrences of jailbreaks, hallucinations, and comprehension
errors. Our work exposes these models' inherent vulnerabilities and challenges
the notion of human-level language comprehension of these models. We have
empirically analysed the impact of non-standard Unicode characters on LLMs and
their safeguarding mechanisms on the best-performing LLMs, including GPT-4,
Gemini 1.5 Pro, LlaMA-3-70B, and Claude 3 Opus. By incorporating alphanumeric
symbols from Unicode outside the standard Latin block and variants of
characters in other languages, we observed a reduction in the efficacy of
guardrails implemented through Reinforcement Learning Human Feedback (RLHF).
Consequently, these models exhibit heightened vulnerability to content policy
breaches and prompt leakage. Our study also suggests a need to incorporate
non-standard Unicode text in LLM training data to enhance the capabilities of
these models.

通过比较分析十五个不同模型的表现，我们揭示了这些模型固有的脆弱性，挑战了它们人类级语言理解的观念，并探讨了非标准 Unicode 字符对 LLMs 的影响及其在最佳性能 LLMs 上的保护机制，包括 GPT-4、Gemini 1.5 Pro、LlaMA-3-70B 和 Claude 3 Opus。