Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Code for fine-tuning Platypus fam LLMs using LoRA
| Date | Stars |
|---|---|
| 2026-07-31 | 625 |
| 2026-08-06 | 625 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Platypus: Quick, Cheap, and Powerful Refinement of LLMs (https://platypus-llm.github.io) <p align="center"> <img src="./assets/Best_Platty.png" alt="Platypus" width="300"/> </p> The Platypus models are a series of fine-tuned and merged variants based on the LLaMA and LLaMa-2 transformer architectures. Platypus takes advantage of [LoRA](https://arxiv.org/pdf/2106.09685.pdf) and [PEFT](https://github.com/huggingface/peft). All models and dataset available via HuggingFace: [`garage-bAInd`](https://huggingface.co/garage-bAInd) ## Updates **8/21/23**: If you're fine-tuning LLaMa-2 7B, please add `bf16=True` and change `fp16=False` in the HF trainer. LLaMa-1 7B works as is. **This only applies to LLaMa-2 7B.** Additionally, if you are using 1 GPU, please change `ddp_find_unused_paramters=False` in the HF trainer. We will be updating the fine-tuning script to handle these changes automatically. **8/14/23**: We have cleaned up our pipeline and added data refinement and similarity code. Within in the next few days we'll have a script to reproduce our exact dataset from 11 open-source datasets. **8/13/23**: An unquantized GPU chatbot of OpenOrca-Platypus2-13B, our most recent collab, is available via Hugging Face spaces, courtesy of OpenOrca: [Chat now!](https://huggingface.co/spaces/Open-Orca/OpenOrca-Platypus2-13B) <p align="center"> <img src="./assets/orca_platty.jpeg" alt="Platypus" width="120"/> </p> **8/11/23**: Our [paper](https://arxiv.org/abs/2308.07317) and [project website](https://platypus-llm.github.io) have been released! ## CLI [Fastchat](https://github.com/lm-sys/FastChat) provides a simple setup for those interested in running the model. After downloading the model through HuggingFace, clone the Fastchat repository: ``` git clone https://github.com/lm-sys/FastChat.git cd FastChat ``` Download the required packages: ``` pip3 install --upgrade pip # enable PEP 660 support pip3 install -e . ``` Finally, run the following: ``` python3 -m fastchat.serve.cli --model-path garage-bAInd/Platypus-30B --conv_template alpaca ``` ## Local Setup This repository is multi-GPU friendly, and provides code to use model or data parellelism, depending on your computational resources. 1. Install dependencies ```bash pip install -r requirements.txt ``` 2. Be sure to use these exact requirements or you may run into model saving or OOM issues. ## Fine-tuning (`finetune.py`) Run `fine-tuning.sh`. Note: The script above uses `torchrun` for data parallelism. PyTorch is not in `requirements.txt` since technically you can run fine-tuning without it (after a few minor changes to the .py file). To use `fine-tuning.sh`, please install [PyTorch](https://pytorch.org/get-started/locally/). We recommend using `torchrun` and PyTorch 2.0+ for speed + `torch.compile`. If you do not install pytorch, or use an alternative method like `accelerate launch`, please take time to comment out any torch related lines in the scirpts. Hyperparameters used to fine-tune Platypus: | Hyperparameter | Value 13B / 70B | |---------------------|--------| | learning rate | 4e-4 / 3e-4 | | batch size | 16 | | microbatch size | 1 | | warmup steps | 100 | | epochs | 1 | | weight decay | 0. | | lr scheduler | cosine | | lora alpha | 16 | | lora rank | 16 | | lora dropout | 0.05 | | lora target modules | gate_proj, up_proj, down_proj| | cutoff length | 4096 | | train on inputs | False | | group by length | False | | add eos token | False | Example for how to calcualte gradient accumulation steps using 2 GPUs: = global_batch_size / micro_batch_size / num_gpus = 16 / 1 / 2 = 8. If your model **cannot** fit on the memory of each GPU, please use the alternative fine-tuning option below (or utilize accelerate, FDSP, etc.) to take advantage of model parallelism. A good alternative to torchrun is accelerate.
Excerpt of 8,328 characters
Read on GitHub
40
4
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:9c41319a36bd63e9, desc:fine-tuning, desc:fine tuning, desc:lora