TokenGate prunes prompt tokens to cut LLM time-to-first-token

Researchers from Stanford and UC Berkeley, Liana Patel, Rishabh Ranjan, Ion Stoica, Matei Zaharia, and Carlos Guestrin, have published a paper on arXiv introducing TokenGate, a small add-on component for existing language models that learns to prune less-important tokens from long inputs before they pass through the model's layers.

The core idea is that models currently process every token in a long input, such as a code repository or lengthy document, through every layer, which is costly. TokenGate instead learns which tokens actually matter and drops the rest during the prefill stage. To make this trainable despite the pruning step normally being non-differentiable, the authors introduce what they call token-gated attention, a mechanism borrowed from mixture-of-experts designs that reweights attention scores by each token's learned importance.

According to the authors' own results, TokenGate achieves 1.4 to 11.3 times faster time-to-first-token and 1.5 to 3.4 times lower peak memory during prefill, with no loss in accuracy across the tasks they tested. Compared with the best existing token-pruning methods, they report up to 6.4 times faster prefill and 2 times lower peak memory. The method works with both standard transformer models like Qwen3 and hybrid-attention models like Qwen3.5, and the authors have released their code.

Why it matters

Faster prefill on long-context inputs could meaningfully cut latency and cost for anyone running LLMs over large codebases or documents.

Source: arXiv