Skip to content

Release Notes

v1.23.4 (vLLM 0.23.0)

Maintenance and feature release on top of v1.23.0 (33 changes). Targets Apps SDK 1.23 and QEfficient release/v1.23.4. It includes everything in v0.23.0 below.

New Features

  • Encoder Cache (EC) support, with documentation and updated examples (#69, #229, #236)
  • Speculative decoding with disaggregated serving (#75)
  • SpD rejection sampler now runs on Numba; triton-cpu is optional, and SPD dependencies are available as an optional extra (#171, #172, #238)
  • KV cache transfer for rank 4 conv and recurrent states in GDN models (#120)
  • Support for versioned Hexagon tools SDK layouts (#156, #196)
  • Python 3.13 support; Python 3.10 dropped (#168, #197)
  • Setup guide for installing vllm-qaic from Apps SDK wheels (#166)

Bug Fixes

  • Fix gptoss models generating junk or null output (#204)
  • Prevent padded decode rows from aliasing active batch slots (#206)
  • Fix async disaggregated prefill CCL slot cleanup (#223)
  • Fix prefill to wait for the oldest exec object when none are free (#231)
  • Patch VllmConfig.max_concurrent_batches for pipeline prefill (#218)
  • Key disaggregated KV hand-off by request id to prevent collisions (#165)
  • Stop defaulting replicate_kv_heads for disaggregated compile (#241)
  • Fix QEfficient compile config forwarding (#233)
  • Fix CCL prefill multimodal input shapes (#118)
  • Skip chunked-prefill assert for vision encoder (#214)
  • Pin encoder cache/compute budget to max_num_batched_tokens for vision encoder (#178)
  • Increase dataPathTimeoutMs (#200)
  • Remove long_prefill_token_threshold (#74)

PyTorch (JIT) Mode

  • Update torch to 2.13 (#132)
  • Hard-code enable_prefix_caching to False (#131)
  • Fix QAIC_VISIBLE_DEVICES usage in examples (#158)

Build and Dependencies

  • Update efficient-transformers to 1.23.4 (#226)
  • Pin mistral_common to v1.11.3 (#228)
  • SDK version set to 1.23 (#135); fix installer/Dockerfile default dependency versions (#142); use hyphenated vllm-qaic in default SDK path (#177)
  • Fix pytest fixture registration for disaggregated tests (#227)

Version Compatibility

vllm-qaic vLLM Apps SDK QEfficient torch (AOT) torch (PYT)
v1.23.4 v0.23.0 1.23 release/v1.23.4 2.7.0+cpu 2.13.0+cpu

Changes since v1.23.0:

  • Apps SDK: 1.22 -> 1.23
  • QEfficient: main -> release/v1.23.4
  • torch (PYT): 2.11.0+cpu -> 2.13.0+cpu; torchvision (PYT): 0.26.0+cpu -> 0.28.0+cpu
  • mistral_common: >= 1.11.3 -> == 1.11.3
  • Python: 3.10-3.12 -> 3.11-3.13
  • arctic-inference is no longer in the default AOT requirements; install it with pip install vllm-qaic[spd] (arctic-inference==0.1.1)
  • Default SDK install path changed from /opt/qti-aic/integrations/vllm_qaic to /opt/qti-aic/integrations/vllm-qaic
  • Unchanged: vLLM v0.23.0, torch (AOT) 2.7.0+cpu, transformers 5.5.4

Pull Requests

Full Changelog: https://github.com/qualcomm/vllm-qaic/compare/v1.23.0...release/r1.23.4

Contributors

Erick Platero, Sanidhya Singal, Apoorva Gokhale, Chaitanya Sachdeva, quic-vargupt, BharathTeja, Vahid Janfaza, quic-mohmeh

v0.23.0 (vLLM 0.23.0)

Upgrades the vLLM-QAIC plugin to track vLLM v0.23.0, with expanded model support, a new speculative decoding proposer, scheduler correctness fixes, and simplified packaging for AOT/PyTorch builds.

AOT Execution Mode

Model Support

  • Fixed Gemma3 multimodal processing for vLLM 0.23
  • Fixed max_pixels resolution handling for Qwen-VL
  • Fixed batch_size handling for Qwen3-235B
  • Simplified the eager/AOT vision-language example to use dict-based config instead of engine args

Speculative Decoding

  • Added DFlash: speculative decoding with a Diffusion LM proposer

Scheduling & Execution

  • Fixed pending exec-object tracking for async prefill scheduling
  • Fixed decode cleanup callback

PyTorch (JIT) Execution Mode

  • Added Rotary Embedding support
  • Added MoE support for vLLM 0.23.0
  • Aligned QaicWorkerPyt profiler with upstream profiler configuration changes

Build & Packaging

  • Upgraded base vLLM dependency to v0.23.0
  • Made RDMA support optional when building UCX
  • Added Rust frontend build support to Docker AOT/PyTorch images, with a configurable --rust-image flag and output wheel filename argument
  • Changed the default triton-cpu install path to a local package directory, and made triton-cpu build opt-in (TRITON_CPU=1) for AOT wheels
  • Fixed the bf16 kernel build for AI200

CI & Developer Tooling

  • Added a pre-commit validation workflow for PRs
  • Ran pre-commit across the full codebase

Pull Requests

Full Changelog: https://github.com/qualcomm/vllm-qaic/compare/release/v0.15.0...release/v0.23.0

v0.15.0.dev3 (Initial Public Release)

Initial vLLM-QAIC plugin for Qualcomm Cloud AI Accelerators. Introduces V1 architecture support with both Ahead-of-Time (AOT) compiled and PyTorch JIT execution modes.

AOT Execution Mode

  • Generic model execution (LLM, MoE, Embedding, Reranker, Score)
  • QPC execution flow and QEFF model compilation
  • CCL support for pipeline prefill
  • Speculative decoding (Ngram, Suffix, Draft)
  • LoRaX adapter serving (Dynamic & Fixed)
  • Disaggregated Serving:
    • Pipeline prefill execution with Tensor Splitting
    • MDP Partitioner support
    • QAIC shared memory connector
    • LMCache connector
    • Disaggregated orchestration package integration
    • Grafana/Prometheus monitoring dashboard
    • LLM-D support
    • Auto head replication
    • KV cache transfer for heterogeneous attention layers
  • Vision-Language Models:
    • 2QPC and 3QPC VLM execution
    • Qwen3VL, Qwen3VLMoE, InternVL, Qwen2.5VL
  • Scheduling & Performance:
    • Scheduler optimization with async scheduling
    • Chunked prefill scheduling (reduces P99 TPOT)

PyTorch (JIT) Execution Mode

  • PyTorch eager mode execution
  • Fused Kernels (RMSNorm, TopK Softmax, Grouped TopK, TopK Sigmoid)

Version Compatibility

vllm-qaic vLLM Apps SDK QEfficient torch (AOT) torch (PYT)
v0.15.0.dev0 v0.15.0 >= 1.22.0 main 2.7.0+cpu 2.10.0+cpu

Known Issues

  • Prefix caching is not yet supported (planned)
  • PYT mode does not support SpD, LoRA, or disaggregated serving
  • best_of > 1 is not supported

Contributors

Ankit Arora, Sanidhya Singal, Xiyue Shi, Shivani Athavale, Erick Platero, Mahesh Balasubramanian, Animesh Kumar, Apoorva Gokhale, Gokkulnath Thirukonda Srinivasan, Jou-An Chen, Varun Gupta, Abdul Rasheed Sahni, Mohit Mehta, Neeharika Gupta, Mrunal Kshirsagar, Aditya Dhananjay Jadhav, Mohit Gupta, Sahil Deshmukh, Vahid Janfaza, Chaitanya Sachdeva, Shriniwas Kulkarni