Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash attention v2.
| Date | Stars |
|---|---|
| 2026-07-31 | 288 |
| 2026-08-06 | 288 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# FMS FSDP - (Pre)Training FMS with FSDP The “fms-fsdp” repo is a companion to the [Foundation Model Stack](https://github.com/foundation-model-stack/foundation-model-stack). The goal of this repo is to provide a (pre)training example to efficiently train FMS models, in particular Llama2 by leveraging native PyTorch features - FSDP for training and SDPA implementation of Flash attention v2. While there are many exemplar repositories that can perform pretraining at scale (e.g., [MegatronLM](), [DeepSpeed]()), this work is what IBM has been doing with PyTorch community on using FSDP for training and how to do that efficiently. It is not meant to be an end-to-end framework for training of models, which includes data preparation (pre), and alignment/tuning of the base model (post). For an end-to-end framework, we would recommend the reader to [OLMo](https://github.com/allenai/OLMo) from AllenAI, which provides datasets, data preprocessing frameworks, leverages FSDP on AMD GPUs for training, and provides a tuning/alignment framework. ## Training throughput benchmarks **_numbers are updated with `torch.compile`, as our fms models are fully compatible with torch compile_** We benchmark the best possible throughput and the strategies we employ in the below table and share the throughput obtained on 128 A100 GPUs as well as 96 H100 GPUs, we use the exact same scripts and configurations for these GPUs. | Model Size | Sharding Strategy | Compile | Activation Checkpointing | Batch Size | Training Throughput <br/> tokens/sec/GPU <br/> A100 80G 128 GPUs with 400Gbps | Training throughput <br/> tokens/sec/GPU <br/> H100 96 GPUs with 800 Gbps | |------------|-------------------|---------|--------------------------|------------|-------------------------------------------------------------------------------|---------------------------------------------------------------------------| | 7b | HSDP | Y | No AC | 2 | 4550 | 9600 | | 13b | FSDP | Y | Selective AC | 2 | 2150 | 4850 | | 34b | FSDP | Y | Selective AC | 2 | 820 | 1830 | | 70b | FSDP | Y | Selective AC | 2 | 410 | 890 | HFU numbers are computed using the [PyTorch FLOP counter](https://github.com/pytorch/pytorch/blob/2240018c03744ee34ea14ad53481db934c37e384/torch/utils/flop_counter.py#L336) and the theoretical bf16 performance of A100 and H100 GPUs, whereas MFU numbers are computed using the methodology outlined in [NanoGPT](https://github.com/karpathy/nanoGPT) and the [PaLM](https://arxiv.org/pdf/2204.02311.pdf) paper. | Model Size | Compile | Batch size | MFU (A100 80G) | HFU (A100 80G) | MFU (H100 80G) | HFU (H100 80G) | |------------|---------|------------|----------------|----------------|----------------|----------------| | 7B | Y | 2 | 0.68 | 0.68 | 0.46 | 0.46 | | 13B | Y | 2 | 0.61 | 0.69 | 0.43 | 0.46 | | 34B | Y | 2 | 0.55 | 0.74 | 0.38 | 0.49 | | 70B | Y | 2 | 0.55 | 0.74 | 0.38 | 0.47 | A few points to note here, on the A100s, we note that for 13B we are not utilizing th
Excerpt of 7,787 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:560f8ff3deed5f7a, topic:pytorch, topic:distributed-training
matched fp:560f8ff3deed5f7a, topic:llm