Getting Started¶
New here?
Start with the Quick Start — zero to first inference in under 5 minutes using Docker.
Choose your path based on your use case:
| Goal | Mode | Start here |
|---|---|---|
| Production serving with maximum throughput | AOT | AOT Installation |
| Development / prototyping with flexibility | PYT (Eager) | PYT Installation |
| Fastest path to first inference | AOT Docker | Quickstart |
Which Mode Should I Use?¶
| Criterion | AOT (Ahead-of-Time) | PYT (Eager) (Experimental) |
|---|---|---|
| Compilation | Models pre-compiled to QPCs via QEfficient | Dynamic execution via torch_qaic |
| Latency | Optimized static graphs | Runtime flexibility |
| Features | Full: SpD, LoRA, disaggregated, multimodal, embedding | Selected VLMs |
| Best for | Production workloads | R&D, new model bring-up |
Recommendation
If you are deploying models to serve production traffic, start with AOT mode. If you are experimenting with new model architectures or need rapid iteration, use PYT mode.
Prerequisites¶
Before installing either mode, ensure you have:
- Qualcomm Cloud AI hardware (Cloud AI 100, Cloud AI 080, or Cloud AI 100 Ultra)
- Linux (Ubuntu 22.04+)
- Qualcomm Cloud AI SDK >= 1.22.0
- Python 3.12
See Prerequisites for detailed setup.