Speculative Decoding¶
Speculative decoding (SpD) accelerates token generation by using a fast proposer to draft candidate tokens that the target model verifies in a single forward pass. When acceptance rates are high, effective tokens-per-second during decode can increase significantly (actual improvement is workload-dependent).
Quick recommendation
Start with ngram SpD — it requires no separate model binary or dedicated core allocation and works well for summarization and conversational workloads. Switch to draft_model if you need higher acceptance rates on diverse generation tasks.
AOT rejection sampler runs on Numba
In AOT mode, the host-side rejection sampler runs on Numba (pinned in the AOT
requirements and installed by ./scripts/install.sh aot). No triton-cpu backend
is needed for SpD or normal AOT serving. The dedicated parity environment must
be installed with TRITON_CPU=1 to run mandatory Tier A Numba/Triton tests.
Methods¶
Cloud AI supports four speculative decoding methods:
| Method | Proposer | Separate Model Binary | Dedicated Core Allocation | Best For |
|---|---|---|---|---|
ngram |
N-gram pattern matching | No | No | Summarization, repetitive output |
suffix |
Suffix array matching | No | No | Code completion, long-context echo |
draft_model |
Lightweight LLM | Yes | Yes | General text generation |
dflash |
Block-diffusion draft LM | Yes | Yes | High-throughput block drafting |
N-gram Speculative Decoding¶
Requires no separate model binary and no dedicated core allocation — runs within the target model's existing resources:
from vllm import LLM, SamplingParams
llm = LLM(
model="meta-llama/Llama-3.1-8B-Instruct",
max_num_seqs=8,
max_model_len=2048,
quantization="mxfp6",
kv_cache_dtype="mxint8",
additional_config={"override_qaic_config": {"prefill_seq_len": 128}},
speculative_config={
"method": "ngram",
"num_speculative_tokens": 5,
},
)
Docker example:
docker run --rm -it --network host \
--device /dev/accel/ \
--shm-size=4gb \
ghcr.io/quic/cloud_ai_inference_vllm:1.21.2.0 \
--host 127.0.0.1 --port 8000 \
--model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
--max-model-len 256 \
--max-num-seq 16 \
--max-seq-len-to-capture 128 \
--quantization mxfp6 \
--kv-cache-dtype mxint8 \
--speculative-config '{"method":"ngram","num_speculative_tokens":5}'
Draft-Model Speculative Decoding¶
Uses a lightweight model (e.g., Llama-3.2-1B) to propose tokens for a larger target model (e.g., Llama-3.1-8B):
from vllm import LLM, SamplingParams
llm = LLM(
model="meta-llama/Llama-3.1-8B-Instruct",
max_num_seqs=4,
max_model_len=2048,
quantization="mxfp6",
kv_cache_dtype="mxint8",
additional_config={
"override_qaic_config": {
"device_group": [0, 1, 2, 3],
"num_cores": 10, # 10 cores for target model
"prefill_seq_len": 128,
},
"draft_override_qaic_config": {
"device_group": [0, 1, 2, 3], # Same device
"num_cores": 6, # 6 cores for draft model
},
},
speculative_config={
"method": "draft_model",
"model": "meta-llama/Llama-3.2-1B-Instruct",
"num_speculative_tokens": 3,
},
)
Docker example:
docker run --rm -it --network host \
--device /dev/accel/ \
--shm-size=4gb \
-e HF_TOKEN=<your_hf_token> \
ghcr.io/quic/cloud_ai_inference_vllm:1.21.2.0 \
--host 127.0.0.1 --port 8000 \
--model meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 2048 \
--max-num-seq 4 \
--max-seq-len-to-capture 128 \
--quantization mxfp6 \
--kv-cache-dtype mxint8 \
--speculative-config '{"method":"draft_model","model":"meta-llama/Llama-3.2-1B-Instruct","num_speculative_tokens":3}' \
--additional-config '{"override_qaic_config":{"device_group":[0,1,2,3],"num_cores":10},"draft_override_qaic_config":{"device_group":[0,1,2,3],"num_cores":6}}'
QEfficient Reference
Draft model compilation requires separate QPC generation. See the QEfficient SpD Guide for compilation details.
DFlash Speculative Decoding¶
DFlash uses a block-diffusion draft LM (DLM) that proposes an entire block of candidate tokens per step in a single batched forward pass, conditioned on the target model's (TLM) hidden states. The DLM's block_size sets num_speculative_tokens (= block_size - 1):
from vllm import LLM, SamplingParams
device_group = [0, 1, 2, 3]
block_size = 16 # DFlash DLM block_size (num_speculative_tokens = block_size - 1)
llm = LLM(
model="Qwen/Qwen3-4B",
max_num_seqs=4,
max_model_len=4096,
quantization="mxfp6",
kv_cache_dtype="mxint8",
additional_config={
"override_qaic_config": {
"device_group": device_group,
"num_cores": 8,
"prefill_seq_len": 128, # TLM prefill_seq_len (multiple of block_size)
"mxfp6_matmul": True,
"mxint8_kv_cache": True,
"mos": 1,
},
"draft_override_qaic_config": {
"device_group": device_group, # Same device
"num_cores": 8,
"prefill_seq_len": block_size, # DLM prefills block_size at a time
"mxfp6_matmul": True,
"mxint8_kv_cache": True,
"mos": 1,
},
},
speculative_config={
"method": "dflash",
"model": "z-lab/Qwen3-4B-DFlash-b16",
"num_speculative_tokens": block_size - 1,
},
)
Docker example:
docker run --rm -it --network host \
--device /dev/accel/ \
--shm-size=4gb \
-e HF_TOKEN=<your_hf_token> \
ghcr.io/quic/cloud_ai_inference_vllm:1.21.2.0 \
--host 127.0.0.1 --port 8000 \
--model Qwen/Qwen3-4B \
--max-model-len 4096 \
--max-num-seq 4 \
--max-seq-len-to-capture 128 \
--quantization mxfp6 \
--kv-cache-dtype mxint8 \
--speculative-config '{"method":"dflash","model":"z-lab/Qwen3-4B-DFlash-b16","num_speculative_tokens":15}' \
--additional-config '{"override_qaic_config":{"device_group":[0,1,2,3],"num_cores":8,"prefill_seq_len":128,"mxfp6_matmul":true,"mxint8_kv_cache":true,"mos":1},"draft_override_qaic_config":{"device_group":[0,1,2,3],"num_cores":8,"prefill_seq_len":16,"mxfp6_matmul":true,"mxint8_kv_cache":true,"mos":1}}'
QEfficient Reference
The DFlash DLM checkpoint (e.g. z-lab/Qwen3-4B-DFlash-b16) and its
cross-checkpoint config (block_size, target_layer_ids, mask_token_id)
are consumed via QEfficient's DFlashDLMTransform/DFlashTLMTransform. See
the QEfficient SpD Guide
for details.
TLM prefill_seq_len must be a multiple of block_size
The target model's prefill_seq_len must be
an exact multiple of the DLM's block_size, since prefill chunks are fanned
out into block_size-sized sub-blocks for the DLM.
Core Allocation¶
On a Cloud AI 100 Ultra card (4 QIDs, 16 cores per QID), cores are split between the target model (TLM) and draft model (DLM) on each QID:
Per QID (16 NSP cores):
+--------------------+----------------+
| Target Model (10) | Draft Model (6)|
+--------------------+----------------+
Applied across QID 0-3 -- both models share the same device group.
No additional hardware is required — both models run on the same device.
Default allocation
When no explicit num_cores configuration is provided, cores are split 8/8 (equal between target and draft). The 10/6 split shown above is a recommended allocation for an 8B target + 1B draft.
Performance Tuning¶
Tuning parameters
| Parameter | Effect | Recommendation |
|---|---|---|
num_speculative_tokens |
Tokens proposed per step | 3-5 (higher = more aggressive) |
override_qaic_config.num_cores |
Compute for verification (target) | 10-12 for 8B models |
draft_override_qaic_config.num_cores |
Compute for proposing (draft) | 4-6 for 1B models |
max_num_seqs |
Batch size | Lower batch = higher acceptance rate |
LD_PRELOAD libiomp5 for CPU-bound SpD proposers (AOT mode)
For AOT mode, ngram/suffix SpD proposers run on host CPU and benefit
from Intel OpenMP (libiomp5.so) runtime tuning — on both Intel and
AMD hosts. Benchmarks on an AMD EPYC host showed meaningful throughput
gains for CPU-bound proposers like ngram, with the improvement growing
at higher batch sizes (actual gains are workload-dependent).
# install Intel OpenMP (provides libiomp5.so) into your active venv
pip install intel-openmp
# manually find the path
IOMP_PATH=$(find "$(python -c 'import sysconfig; print(sysconfig.get_paths()["data"])')" -iname "libiomp5.so" | head -1)
# add it to LD_PRELOAD
export LD_PRELOAD="$IOMP_PATH:$LD_PRELOAD"
When libiomp5.so is on LD_PRELOAD, vllm-qaic sets KMP_BLOCKTIME=1 and
KMP_TPAUSE=0. Set VLLM_DISABLE_LD_PRELOAD_OPT=1 to opt out of this tuning.
SpD with Disaggregated Serving¶
SpD can be combined with disaggregated serving — proposals are generated and verified entirely on the decode node. See Disaggregated Serving.