Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Guide: Finetune GPT2-XL (1.5 Billion Parameters) and finetune GPT-NEO (2.7 B) on a single GPU with Huggingface Transformers using DeepSpeed
| Date | Stars |
|---|---|
| 2026-07-31 | 435 |
| 2026-08-01 | 435 |
| 2026-08-02 | 435 |
| 2026-08-06 | 435 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Guide: Finetune GPT2-XL (1.5 Billion Parameters) and GPT-NEO (2.7 Billion Parameters) on a single GPU with Huggingface Transformers using DeepSpeed - Finetuning large language models like GPT2-xl is often difficult, as these models are too big to fit on a single GPU. - This guide explains how to finetune GPT2-xl and GPT-NEO (2.7B Parameters) with just one command of the Huggingface Transformers library on a single GPU. - This is made possible by using the DeepSpeed library and gradient checkpointing to lower the required GPU memory usage of the model. - I also explain how to set up a server on Google Cloud with a V100 GPU (16GB VRAM), that you can use if you don't have a GPU with enough VRAM (16+ GB) or you don't have enough enough normal RAM (60 GB+). ## 1. (Optional) Setup VM with V100 in Google Compute Engine Note: The GPT2-xl model does run on any server with a GPU with at least 16 GB VRAM and 60 GB RAM. The GPT-NEO model needs at least 70 GB RAM. If you use your own server and not the setup described here, you will need to install CUDA and Pytorch on it. ### Requirements 1. Install the Google Cloud SDK: [Click Here](https://cloud.google.com/sdk/docs/install) 2. Register a Google Cloud Account, create a project and set up billing (only once you set up billing, you can use the $300 dollar sign up credit for GPUs). 3. Request a quota limit increase for "GPU All Regions" to 1. [Here](https://nirbenz.github.io/gce-quota/) is a step by step guide. The UI changed a bit and looks now like [this](https://stackoverflow.com/a/62883281/15447124). 4. Log in and initialize the cloud sdk with `gcloud auth login` and `gcloud init` and follow the steps until you are set up. ### Create VM - Replace YOURPROJECTID in the command below with the project id from your GCE project. - You can remove the `--preemptible` flag from the command below, but keeping it reduces your cost to about 1/3 and allows Google to shut down your instance at any point. At the time of writing, this configuration only costs about $1.28 / hour in GCE, when using preemptible. Depending on the size of your dataset, finetuning usually only takes a few hours. - You can change the zone, if there are no ressources available. [Here](https://cloud.google.com/compute/docs/gpus/gpu-regions-zones) is a list of all zones and whether they have V100 GPUs. Depending on the time of the day you might need to try out a few. Usually there are also more server available if you keep the `--preemptible` flag - We need a GPU server with at least 60 GB RAM, otherwise the run will crash, whenever the script wants to save/pickle a model. This setup below gives us as much RAM as possible with 12 CPU cores in GCE (without paying for extended memory). You also can't use more than 12 CPU cores with a single V100 GPU in GCE. Run this to create the instance: ```markdown gcloud compute instances create gpuserver \ --project YOURPROJECTID \ --zone us-west1-b \ --custom-cpu 12 \ --custom-memory 78 \ --maintenance-policy TERMINATE \ --image-family pytorch-1-7-cu110 \ --image-project deeplearning-platform-release \ --boot-disk-size 200GB \ --metadata "install-nvidia-driver=True" \ --accelerator="type=nvidia-tesla-v100,count=1" \ --preemptible ``` After 5 minutes or so (the server needs to install nvidia drivers first), you can connect to your instance with the command below. If you changed the zone, you also will need to change it here. - replace YOURSDKACCOUNT with your sdk account name ```markdown gcloud compute ssh YOURSDKACCOUNT@gpuserver --zone=us-west1-b ``` Don't forget to shut down the server once your done, otherwise you will keep getting billed for it. This can be done [here](https://console.cloud.google.com/compute/instance). The next time you can restart the server from the same web ui [here](https://console.cloud.google.com/compute/instance). ## 2. Download script and install libraries Run this to download the script and to install all li
Excerpt of 12,482 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:30caa3dad22d5824, topic:finetuning, name:finetune, desc:finetune
matched fp:30caa3dad22d5824, topic:deepspeed