Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Training LLMs with QLoRA + FSDP
| Date | Stars |
|---|---|
| 2026-07-31 | 1550 |
| 2026-08-06 | 1550 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# fsdp_qlora Training LLMs with Quantized LoRA + FSDP. Read our [announcement blog post](https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html). You should treat this script as an alpha/preview release. If you’re not comfortable with testing and debugging models, we’d suggest holding off for a few months while the community more fully tests the approach. ## Integrations FSDP+QLoRA has been integrated into: - [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl/pull/1378): experimental support ## Installation The following steps should work (tested on Cuda 11.7, 11.8 and 12.1): - Clone https://github.com/AnswerDotAI/fsdp_qlora - `pip install llama-recipes fastcore "transformers!=4.38.*,!=4.39.*" --extra-index-url https://download.pytorch.org/whl/test/cu118` as an easy way to get most dependencies (replace 118 with your desired Cuda version) - Install bitsandbytes `pip install bitsandbytes>=0.43.0` - Run `huggingface-cli login` (to access Llama 2) - Optional Libraries: - HQQ quantization: follow the HQQ installation [instructions](https://github.com/mobiusml/hqq?tab=readme-ov-file#installation). Our training script uses `HQQBackend.ATEN_BACKPROP`, so also make sure to build the custom kernels `cd hqq/kernels && python setup_cuda.py install`. - Weights and Biases logging: `pip install wandb` - [Pytorch >= 2.2](https://pytorch.org/blog/pytorch2-2/) is recommended to make use of the native flash-attention 2 kernel. ## Finetune Llama-2 70B on Dual 24GB GPUs Once installed, run `cd fsdp_qlora` and then run the following command to begin finetuning Llama-2 70B on [Alpaca](https://huggingface.co/datasets/yahma/alpaca-cleaned) at a maximum sequence length of 512 tokens. ```bash python train.py \ --model_name meta-llama/Llama-2-70b-hf \ --batch_size 2 \ --context_length 512 \ --precision bf16 \ --train_type qlora \ --use_gradient_checkpointing true \ --use_cpu_offload true \ --dataset alpaca \ --reentrant_checkpointing true ``` This example command currently uses just over 128GB of CPU RAM. If you only have 128GB available, we recommend making a 10-20GB swap file to accommodate the initial spike in usage. ## Training Options For quantization we support HQQ and bitsandbytes. We're currently doing benchmarking to help you decide which to use. If you do use bitsandbytes, be sure to pass `--reentrant_checkpointing True` to avoid triggering a bug in bitsandbytes which results in high memory usage (a fix is in progress). ### `--train_type full` Full params fine-tuning. ```bash export CUDA_VISIBLE_DEVICES=4,5 # optionally set devices python train.py \ --world_size 2 \ # optional, on a single machine will be set automatically --master_port 12356 \ # optional, defaults to 12355 --model_name meta-llama/Llama-2-7b-hf \ --gradient_accumulation_steps 4 \ --batch_size 8 \ --context_length 512 \ --precision bf16 \ --train_type full \ --use_gradient_checkpointing true \ --use_cpu_offload false \ --use_activation_cpu_offload false \ --log_to wandb \ --dataset alpaca ``` ### `--train_type lora` LoRA fine-tuning using HF PEFT library. ```diff - --train_type full \ + --train_type lora \ ``` ### `--train_type custom_lora` LoRA fine-tuning using a custom LoRA module. ```diff - --train_type full \ + --train_type custom_lora \ ``` ### `--train_type qlora` 4-bit quantized LoRA fine-tuning using bitsanbytes Linear4bit layer with NF4 quantization and HF PEFT library. ```diff - --train_type full \ + --train_type qlora \ + --reentrant_checkpointing true \ ``` ### `--train_type custom_qlora` 4-bit quantized LoRA fine-tuning using bitsanbytes Linear4bit layer with NF4 quantization and a custom LoRA module. ```diff - --train_type full \ + --train_type custom_qlora \ + --reentrant_checkpointing true \ ``` ### `--train_type hqq_lora` 4-bit quantized LoRA fine-tuning using HQQ library and a custom LoRA module. ```diff - --train_type full \ + --train_type hqq_lora \ ``` ### `--train_type bnb_dora` 4-bit quantized DoRA fine-tun
Excerpt of 13,483 characters
Read on GitHub
88
Jonathan Whitaker · United States
49
Benjamin Warner
48
15
Austin Huang
6
4
Geronimo
2
Jeremy Howard · @answerdotai · Australia
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:ee90877c5374d343, name:qlora, desc:qlora