Skip to content

Speculative Decoding

Speculative decoding (SpD) accelerates token generation by using a fast proposer to draft candidate tokens that the target model verifies in a single forward pass. When acceptance rates are high, effective tokens-per-second during decode can increase significantly (actual improvement is workload-dependent).

Quick recommendation

Start with ngram SpD — it requires no separate model binary or dedicated core allocation and works well for summarization and conversational workloads. Switch to draft_model if you need higher acceptance rates on diverse generation tasks.

AOT rejection sampler runs on Numba

In AOT mode, the host-side rejection sampler runs on Numba (pinned in the AOT requirements and installed by ./scripts/install.sh aot). No triton-cpu backend is needed for SpD or normal AOT serving. The dedicated parity environment must be installed with TRITON_CPU=1 to run mandatory Tier A Numba/Triton tests.

Methods

Cloud AI supports four speculative decoding methods:

Method Proposer Separate Model Binary Dedicated Core Allocation Best For
ngram N-gram pattern matching No No Summarization, repetitive output
suffix Suffix array matching No No Code completion, long-context echo
draft_model Lightweight LLM Yes Yes General text generation
dflash Block-diffusion draft LM Yes Yes High-throughput block drafting

N-gram Speculative Decoding

Requires no separate model binary and no dedicated core allocation — runs within the target model's existing resources:

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    max_num_seqs=8,
    max_model_len=2048,
    quantization="mxfp6",
    kv_cache_dtype="mxint8",
    additional_config={"override_qaic_config": {"prefill_seq_len": 128}},
    speculative_config={
        "method": "ngram",
        "num_speculative_tokens": 5,
    },
)

Docker example:

docker run --rm -it --network host \
  --device /dev/accel/ \
  --shm-size=4gb \
  ghcr.io/quic/cloud_ai_inference_vllm:1.21.2.0 \
  --host 127.0.0.1 --port 8000 \
  --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
  --max-model-len 256 \
  --max-num-seq 16 \
  --max-seq-len-to-capture 128 \
  --quantization mxfp6 \
  --kv-cache-dtype mxint8 \
  --speculative-config '{"method":"ngram","num_speculative_tokens":5}'

Draft-Model Speculative Decoding

Uses a lightweight model (e.g., Llama-3.2-1B) to propose tokens for a larger target model (e.g., Llama-3.1-8B):

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    max_num_seqs=4,
    max_model_len=2048,
    quantization="mxfp6",
    kv_cache_dtype="mxint8",
    additional_config={
        "override_qaic_config": {
            "device_group": [0, 1, 2, 3],
            "num_cores": 10,       # 10 cores for target model
            "prefill_seq_len": 128,
        },
        "draft_override_qaic_config": {
            "device_group": [0, 1, 2, 3],   # Same device
            "num_cores": 6,        # 6 cores for draft model
        },
    },
    speculative_config={
        "method": "draft_model",
        "model": "meta-llama/Llama-3.2-1B-Instruct",
        "num_speculative_tokens": 3,
    },
)

Docker example:

docker run --rm -it --network host \
  --device /dev/accel/ \
  --shm-size=4gb \
  -e HF_TOKEN=<your_hf_token> \
  ghcr.io/quic/cloud_ai_inference_vllm:1.21.2.0 \
  --host 127.0.0.1 --port 8000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 2048 \
  --max-num-seq 4 \
  --max-seq-len-to-capture 128 \
  --quantization mxfp6 \
  --kv-cache-dtype mxint8 \
  --speculative-config '{"method":"draft_model","model":"meta-llama/Llama-3.2-1B-Instruct","num_speculative_tokens":3}' \
  --additional-config '{"override_qaic_config":{"device_group":[0,1,2,3],"num_cores":10},"draft_override_qaic_config":{"device_group":[0,1,2,3],"num_cores":6}}'

QEfficient Reference

Draft model compilation requires separate QPC generation. See the QEfficient SpD Guide for compilation details.

DFlash Speculative Decoding

DFlash uses a block-diffusion draft LM (DLM) that proposes an entire block of candidate tokens per step in a single batched forward pass, conditioned on the target model's (TLM) hidden states. The DLM's block_size sets num_speculative_tokens (= block_size - 1):

from vllm import LLM, SamplingParams

device_group = [0, 1, 2, 3]
block_size = 16  # DFlash DLM block_size (num_speculative_tokens = block_size - 1)

llm = LLM(
    model="Qwen/Qwen3-4B",
    max_num_seqs=4,
    max_model_len=4096,
    quantization="mxfp6",
    kv_cache_dtype="mxint8",
    additional_config={
        "override_qaic_config": {
            "device_group": device_group,
            "num_cores": 8,
            "prefill_seq_len": 128,  # TLM prefill_seq_len (multiple of block_size)
            "mxfp6_matmul": True,
            "mxint8_kv_cache": True,
            "mos": 1,
        },
        "draft_override_qaic_config": {
            "device_group": device_group,  # Same device
            "num_cores": 8,
            "prefill_seq_len": block_size,  # DLM prefills block_size at a time
            "mxfp6_matmul": True,
            "mxint8_kv_cache": True,
            "mos": 1,
        },
    },
    speculative_config={
        "method": "dflash",
        "model": "z-lab/Qwen3-4B-DFlash-b16",
        "num_speculative_tokens": block_size - 1,
    },
)

Docker example:

docker run --rm -it --network host \
  --device /dev/accel/ \
  --shm-size=4gb \
  -e HF_TOKEN=<your_hf_token> \
  ghcr.io/quic/cloud_ai_inference_vllm:1.21.2.0 \
  --host 127.0.0.1 --port 8000 \
  --model Qwen/Qwen3-4B \
  --max-model-len 4096 \
  --max-num-seq 4 \
  --max-seq-len-to-capture 128 \
  --quantization mxfp6 \
  --kv-cache-dtype mxint8 \
  --speculative-config '{"method":"dflash","model":"z-lab/Qwen3-4B-DFlash-b16","num_speculative_tokens":15}' \
  --additional-config '{"override_qaic_config":{"device_group":[0,1,2,3],"num_cores":8,"prefill_seq_len":128,"mxfp6_matmul":true,"mxint8_kv_cache":true,"mos":1},"draft_override_qaic_config":{"device_group":[0,1,2,3],"num_cores":8,"prefill_seq_len":16,"mxfp6_matmul":true,"mxint8_kv_cache":true,"mos":1}}'

QEfficient Reference

The DFlash DLM checkpoint (e.g. z-lab/Qwen3-4B-DFlash-b16) and its cross-checkpoint config (block_size, target_layer_ids, mask_token_id) are consumed via QEfficient's DFlashDLMTransform/DFlashTLMTransform. See the QEfficient SpD Guide for details.

TLM prefill_seq_len must be a multiple of block_size

The target model's prefill_seq_len must be an exact multiple of the DLM's block_size, since prefill chunks are fanned out into block_size-sized sub-blocks for the DLM.

Core Allocation

On a Cloud AI 100 Ultra card (4 QIDs, 16 cores per QID), cores are split between the target model (TLM) and draft model (DLM) on each QID:

Per QID (16 NSP cores):
+--------------------+----------------+
| Target Model (10)  | Draft Model (6)|
+--------------------+----------------+

Applied across QID 0-3 -- both models share the same device group.

No additional hardware is required — both models run on the same device.

Default allocation

When no explicit num_cores configuration is provided, cores are split 8/8 (equal between target and draft). The 10/6 split shown above is a recommended allocation for an 8B target + 1B draft.

Performance Tuning

Tuning parameters
Parameter Effect Recommendation
num_speculative_tokens Tokens proposed per step 3-5 (higher = more aggressive)
override_qaic_config.num_cores Compute for verification (target) 10-12 for 8B models
draft_override_qaic_config.num_cores Compute for proposing (draft) 4-6 for 1B models
max_num_seqs Batch size Lower batch = higher acceptance rate

LD_PRELOAD libiomp5 for CPU-bound SpD proposers (AOT mode)

For AOT mode, ngram/suffix SpD proposers run on host CPU and benefit from Intel OpenMP (libiomp5.so) runtime tuning — on both Intel and AMD hosts. Benchmarks on an AMD EPYC host showed meaningful throughput gains for CPU-bound proposers like ngram, with the improvement growing at higher batch sizes (actual gains are workload-dependent).

# install Intel OpenMP (provides libiomp5.so) into your active venv
pip install intel-openmp

# manually find the path
IOMP_PATH=$(find "$(python -c 'import sysconfig; print(sysconfig.get_paths()["data"])')" -iname "libiomp5.so" | head -1)

# add it to LD_PRELOAD
export LD_PRELOAD="$IOMP_PATH:$LD_PRELOAD"

When libiomp5.so is on LD_PRELOAD, vllm-qaic sets KMP_BLOCKTIME=1 and KMP_TPAUSE=0. Set VLLM_DISABLE_LD_PRELOAD_OPT=1 to opt out of this tuning.

SpD with Disaggregated Serving

SpD can be combined with disaggregated serving — proposals are generated and verified entirely on the decode node. See Disaggregated Serving.