Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
MU-LLaMA: Music Understanding Large Language Model
| Date | Stars |
|---|---|
| 2026-07-31 | 306 |
| 2026-08-03 | 307 |
| 2026-08-06 | 307 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<div>
<h1>
MU-LLaMA: <br>Music Understanding Large Language Model
<img src="./assets/logo.png" height=100px align="right"/>
</h1>
</div>
[](https://arxiv.org/abs/2308.11276)
[]([https://arxiv.org/abs/2308.11276](https://huggingface.co/datasets/mu-llama/MusicQA))
[](https://paperswithcode.com/sota/music-question-answering-on-musicqa-dataset?p=music-understanding-llama-advancing-text-to)
This is the official repository for *[Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning](https://arxiv.org/abs/2308.11276)*
The demo page with more information regarding the MU-LLaMA model is avilable [here](https://crypto-code.github.io/MU-LLaMA-Demo/).
## Introduction
The MU-LLaMA model is Music Understanding Language Model designed with the purpose of answering questions based on music. Our model is also designed with the purpose of captioning music files to generate Text-to-Music Generation datasets. The model uses MERT + LLaMA as the backbone and employs an adapter to encoperate music context information to guide LLaMA's output. MERT was chosen as the music encoder for our model after comparison of different music representation models, which can be viewed [here](https://github.com/crypto-code/Music-Representation-Comparison). We also provide the code for generating our MusicQA dataset from [MusicCaps](https://www.kaggle.com/datasets/googleai/musiccaps) and the [MagnaTagATune](https://mirg.city.ac.uk/codeapps/the-magnatagatune-dataset) datasets.
<p align="center">
<img src="./assets/MU-LLaMA.png">
</p>
## MU-LLaMA Demo
For the working of our model, Facebook's LLaMA-2 model weights are required, details on obtaining these weights are given on [HuggingFace](https://huggingface.co/docs/transformers/main/model_doc/llama). Our pretrained weights for the MU-LLaMA model, finetuned from **LLaMA 7B-2** can be downloaded [here](https://huggingface.co/mu-llama/MU-LLaMA/tree/main). Once downloaded, store the files in the ckpts folder within the MU-LLaMA directory.
Once downloaded the directory structure will be as shown below.
```
.
├── ...
├── MU-LLaMA
│ ├── ckpts
│ │ │── LLaMA
│ │ │ │── 7B
│ │ │ │ │── checklist.chk
│ │ │ │ │── consolidated.00.pth
│ │ │ │ │── params.json
│ │ │ │── llama.sh
│ │ │ │── tokenizer.model
│ │ │ │── tokenizer_checklist.chk
│ │ │── 7B.pth
│ │ ├── checkpoint.pth
└── ...
```
We use Python 3.9.17 for this project and the library requirements are given in [***requirements.txt***](./requirements.txt). The demo can be run using [***gradio_app.py***](./MU-LLaMA/gradio_app.py).
```
python gradio_app.py --model ./ckpts/checkpoint.pth --llama_dir ./ckpts/LLaMA
```
## Training MU-LLaMA
To train the MU-LLaMA model, follow the steps as below.
### MusicQA Dataset
We use the [MusicCaps](https://www.kaggle.com/datasets/googleai/musiccaps) and the [MagnaTagATune](https://mirg.city.ac.uk/codeapps/the-magnatagatune-dataset) dataset to generate our training MusicQA dataset and the [MTG-Jamendo](https://github.com/MTG/mtg-jamendo-dataset) for evaluation. You can download the generated MusicQA dataset [here](https://huggingface.co/datasets/mu-llama/MusicQA).
To generate the dataset yourself, first download the MusicCaps, MTT and MTG datasets. Once downloaded, the directory structure would be as shown.
```
.
├── ...
├── MusicQA
│ ├── MTT
│ │ ├── audios
│ │ │ │── ...
│ │ ├── annotations_final.csv
│ ├── MusicCaps
│ │ ├── audios
│ │ │ │── ...
│ │ ├── musiccaps-public.csv
│ ├── MTG
│ │ ├── audios
│ │ │ │── 00
│ │ │Excerpt of 10,627 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:504960aca895487e, llm:Repository title and description: 'MU-LLaMA: Music Understanding Large Language Model' (Python project).
matched fp:504960aca895487e, llm:Repository title and description: 'MU-LLaMA: Music Understanding Large Language Model' (Python project).
matched fp:504960aca895487e, llm:Repository title and description: 'MU-LLaMA: Music Understanding Large Language Model' (Python project).
matched fp:504960aca895487e, llm:Repository title and description: 'MU-LLaMA: Music Understanding Large Language Model' (Python project).