The growing scale of Large Language Models (LLMs) has exacerbated inference latency and computational costs. Speculative decoding methods, which aim to mitigate these issues, often face inefficiencies in the construction of token trees and the verification of candidate tokens. Existing strategies, including chain mode, static tree, and dynamic tree approaches, have limitations in accurately preparing candidate token trees for verification. We propose a novel method named C2T that adopts a lightweight classifier to generate and prune token trees dynamically. Our classifier considers additional feature variables beyond the commonly used joint probability to predict the confidence score for each draft token to determine whether it is the candidate token for verification. This method outperforms state-of-the-art (SOTA) methods such as EAGLE-2 on multiple benchmarks, by reducing the total number of candidate tokens by 25% while maintaining or even improving the acceptance length.

本研究针对当前推测解码方法在构建令牌树和验证候选令牌方面的低效问题，提出了一种新颖的C2T方法。该方法利用轻量级分类器动态生成和修剪令牌树，从而在多个基准测试中显著提高性能，减少了候选令牌总数25%，同时保持或改善了接受长度，展现出其潜在的影响力。

C2T：一种用于推测解码的分类器基础树构建方法