Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Latency and Memory Analysis of Transformer Models for Training and Inference
| Date | Stars |
|---|---|
| 2026-07-31 | 490 |
| 2026-08-06 | 491 |
Today
+1 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# llm-analysis
[](https://pypi.org/project/llm-analysis/)
[](https://llm-analysis.readthedocs.io/)
[](https://github.com/cli99/llm-analysis/actions?workflow=tests)
[](https://codecov.io/gh/cli99/llm-analysis)
[](https://github.com/cli99/llm-analysis/blob/main/LICENSE)
> Latency and Memory Analysis of Transformer Models for Training and Inference
- [llm-analysis](#llm-analysis)
- [Overview](#overview)
- [Examples](#examples)
- [Quick Start](#quick-start)
- [Using the `LLMAnalysis` class](#using-the-llmanalysis-class)
- [Using the Entry Point Functions for Command Line](#using-the-entry-point-functions-for-command-line)
- [How to Set FLOPS and Memory Efficiency](#how-to-set-flops-and-memory-efficiency)
- [Current Scope and Limitations](#current-scope-and-limitations)
- [Parallelism Scheme](#parallelism-scheme)
- [Communication](#communication)
- [Activation Recomputation](#activation-recomputation)
- [Data Types](#data-types)
- [Fine-Tuning](#fine-tuning)
- [Assumptions in Inference](#assumptions-in-inference)
- [TODOs (stay tuned :radio:)](#todos-stay-tuned-radio)
- [Citation](#citation)
- [Contributing](#contributing)
- [Useful Links](#useful-links)
## Overview
Many formulas or equations are floating around in papers, blogs, etc., about how to calculate training or inference latency and memory for Large Language Models (LLMs) or Transformers. Rather than doing math on papers or typing in Excel sheets, `let's automate the boring stuff with llm-analysis` :gear:!
Given the specified model, GPU, data type, and parallelism configurations, llm-analysis estimates the latency and memory usage of LLMs for training or inference. With llm-analysis, one can easily try out different training/inference setups theoretically, and better understand the system performance for different scenarios.
llm-analysis helps answer questions such as:
- what batch size, data type, parallelism scheme to use to get a `feasible` (not getting OOM) and `optimal` (maximizing throughput with a latency constraint) setup for training or inference
- `time` it takes with the given setup to do training or inference and the `cost` (GPU-hours)
- how the latency/memory changes if using a different model, GPU type, number of GPU, data type for weights and activations, parallelism configuration (suggesting the performance benefit of `modeling change`, `hardware improvement`, `quantization`, `parallelism`, etc.)
### Examples
Check the example use cases. With llm-analysis, you can do such analysis in minutes :rocket:!
- [Llama 2 Models](examples/llama2)
- [LLaMA Models](examples/llama)
- [Megatron-LM Models](examples/megatron-lm/)
- [Megatron Turing NLG 530B](examples/megatron-turing-nlg)
- [FasterTransformer](examples/fastertransformer/)
## Quick Start
- To install llm-analysis from pypi:
```sh
pip install llm-analysis
```
- To install the latest development build:
```sh
pip install --upgrade git+https://github.com/cli99/llm-analysis.git@main
```
- To install from source, clone the repo and run `pip install .` or `poetry install` (install [poetry](https://python-poetry.org/) by `pip install poetry`).
### Using the `LLMAnalysis` class
To integrate llm-analysis in your code, use the `LLMAnalysis` class. Refer to doc [LLMAnalysis](https://llm-analysis.readthedocs.io/) for details.
`LLMAnalysis` is constructed with flops and memory efficiency numbers and the following [configuration classes](https://llm-analysis.readthedocs.io/en/latest/config.html):
- `ModelConfig` covers model information, i.e. max sequence length, number of transformer layers, nExcerpt of 16,061 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:4a57d54abde0f161, topic:deep-learning
matched fp:4a57d54abde0f161, topic:language-model
matched fp:4a57d54abde0f161, topic:nlp