| Website | Documentation | Contribute | Contact | Changelog |
- [In progress] Porting the entire simulator to Rust, including an in-house network backend replacing ASTRA-Sim, for faster simulation, improved communication-model accuracy, and broader network topology support.
- [In progress] Expanding end-to-end validation for existing model catalogs — DeepSeek, GLM, Kimi, MiniMax, Qwen3.8, and more.
- [2026/10] vLLM 0.31.0 / CUDA 13 update — ~7x faster end-to-end profiling, Kimi-K3 model catalog, bug fixes, and improved Llama/Qwen accuracy with refreshed profiles and benchmarks. (#77)
- [2026/10] vLLM 0.28.0 compatibility update — added DeepSeek, GLM, MiniMax, and Qwen hybrid-model catalogs, fixed bugs, and improved simulation accuracy, validated against the RTXPRO6000 Llama and Qwen benchmark examples. (#76)
- [2026/08] Simulation is ~11x faster with byte-identical results — the four
bench/examplesruns go 16m 40s → 1m 26s. (#67) - [2026/08] TP / PP / EP / DP run in every combination, checked by 58 recorded scenarios (
serving/validate.sh). (#68) - [2026/08] RTX 4090 joins the profile library — within 1% of a real vLLM run on mean TTFT / TPOT / latency. (#59)
See the changelog for v1.3.0 changes, validation scope and upgrade requirements, alongside earlier release notes.
LLMServingSim is a cycle-level simulator for LLM serving infrastructure. It pairs a Python frontend that mirrors vLLM's continuous-batching scheduler with the ASTRA-Sim C++ analytical network backend, and drives both from per-hardware latency data captured by a vLLM-based layerwise profiler. The result is a unified environment for studying heterogeneous accelerators, disaggregated memory tiers (CPU / CXL / PIM), MoE routing, speculative decoding, and multi-instance parallelism (TP / PP / EP / DP) end-to-end — across dense, MoE, sparse-attention and hybrid linear-attention model families.
Multi-rank instance completion follows the slowest rank in each workload iteration, while the first rank may continue to admit PP lookahead work.
git clone --recurse-submodules https://github.com/casys-kaist/LLMServingSim.git
cd LLMServingSim
./scripts/docker-sim.sh # launch the simulator container
./scripts/compile.sh # build ASTRA-Sim + Chakra
./serving/run.sh # run the example simulationsFor installation, configuration, and examples, see the documentation. Existing users should follow the update instructions to synchronize submodules and rebuild ASTRA-Sim and Chakra.
Simulation and comparison against recorded benchmarks are CPU-only. GPU profiling and new benchmarks target vLLM 0.31.0 / CUDA 13.
CLI references: simulator, profiler, bench, and workload generator. See the profile bundle guide for model-specific support and profiling requirements, and validation results for measured accuracy and benchmark configurations.
ISPASS 2026
LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure
Jaehong Cho*, Hyunmin Choi*, Guseul Heo, Jongse Park (KAIST) [Paper]
*Equal contribution
CAL 2025
LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructure
Jaehong Cho, Hyunmin Choi, Jongse Park (KAIST) [Paper]
IISWC 2024
LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale
Jaehong Cho, Minsu Kim, Hyunmin Choi, Guseul Heo, Jongse Park (KAIST) [Paper]
If you use LLMServingSim in your research, please cite:
@INPROCEEDINGS{11527300,
author={Cho, Jaehong and Choi, Hyunmin and Heo, Guseul and Park, Jongse},
booktitle={2026 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)},
title={{LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure}},
year={2026},
pages={1-14},
doi={10.1109/ISPASS69572.2026.00012}
}
@ARTICLE{11224567,
author={Cho, Jaehong and Choi, Hyunmin and Park, Jongse},
journal={IEEE Computer Architecture Letters},
title={{LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving
Techniques in LLM Infrastructure}},
year={2025},
volume={24},
number={02},
pages={361-364},
doi={10.1109/LCA.2025.3628325},
ISSN={1556-6064},
publisher={IEEE Computer Society},
address={Los Alamitos, CA, USA},
month=jul
}
@INPROCEEDINGS{10763697,
author={Cho, Jaehong and Kim, Minsu and Choi, Hyunmin and Heo, Guseul and Park, Jongse},
booktitle={2024 IEEE International Symposium on Workload Characterization (IISWC)},
title={{LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving
at Scale}},
year={2024},
pages={15-29},
doi={10.1109/IISWC63097.2024.00012}
}