Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Implementation of MusicLM, a text to music model published by Google Research, with a few modifications.
| Date | Stars |
|---|---|
| 2026-07-24 | 560 |
| 2026-07-25 | 560 |
| 2026-07-28 | 560 |
| 2026-07-30 | 560 |
| 2026-08-06 | 560 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Open MusicLM Pytorch implementation of [MusicLM](https://arxiv.org/abs/2301.11325), a SOTA text to music model published by Google, with a few modifications. We use [CLAP](https://github.com/LAION-AI/CLAP) as a replacement for MuLan, [Encodec](https://github.com/facebookresearch/encodec) as a replacement for SoundStream, and [MERT](https://huggingface.co/m-a-p/MERT-v0) as a replacement for w2v-BERT. <p align='center'> <img alt='diagram of MusicLM' src='musiclm.png' title="MusicLM" height='250px'> <img alt='diagram of CLAP' src='clap.png' title="CLAP" height='250px'> </p> ## Why CLAP? CLAP is a joint audio-text model trained on [LAION-Audio-630K](https://github.com/LAION-AI/audio-dataset). Similar to MuLan, it consists of an audio tower and a text tower that project their respective media onto a shared latent space (512 dimensions in CLAP vs 128 dimensions in MuLan). MuLan was trained on 50 million text-music pairs. Unfortunately I don't have the data to replicate this, so I'm relying on CLAP's pretrained checkpoints to come close. CLAP was trained on 2.6 million total text-audio pairs from LAION-630k (~633k text-audio pairs) and AudioSet (2 million samples with captions generated by a keyword-to-caption model). Although this is a fraction of the data used to train MuLan, we have successfully used CLAP to generate diverse music samples, which you can listen to [here](https://drive.google.com/drive/folders/1pGY8EP2EZlE2pPpXn5E3YAkoCgATWw3_) (keep in mind these are very early results). In the event that CLAP's latent space is not expressive enough for music generation, we can train CLAP on music or substitute the model for @lucidrain's [MuLan implementation](https://github.com/lucidrains/musiclm-pytorch) once it is trained. ## Why Encodec? SoundStream and Encodec are both neural audio codecs that encode any waveform to a sequence of acoustic tokens, which can then be decoded into a waveform resembling the original. These intermediate tokens can then be modeled as a seq2seq task. [Encodec](https://github.com/facebookresearch/encodec) is released by Facebook and pretrained checkpoints are publicly available, whereas this is not the case with SoundStream. ## Differences from @lucidrains implementation - Autoregressively models the CLAP/MuLan conditioning signal by passing it into the transformers as discrete tokens, as mentioned in section 3.1 of the paper. Musiclm-pytorch conditions on them with cross attention. - TokenConditionedTransformer can support variable token sequences, which makes it easy to do further experimentation (e.g. combining multiple conditioning signals, stereo waveform generation, etc.) - Uses existing open source models instead of training MuLan and SoundStream. - Some modifications to increase the chance of successfully training the model. # End Goal The goal of this project is to replicate the results of MusicLM as quickly as possible without necessarily sticking to the architecture in the paper. For those looking for a more true-to-form implementation, check out [musiclm-pytorch](https://github.com/lucidrains/musiclm-pytorch). We also seek to gain a better understanding of CLAP's latent space. Join us on discord if you'd like to get involved! [<img alt="join discord" src="https://img.shields.io/discord/1079520916591104000?color=%237289da&logo=discord"/>](https://discord.gg/jN8jADShX5) # Usage ## Install ```shell conda env create -f environment.yaml conda activate open-musiclm ``` ## Configs A "model config" contains information about the model architecture such as the number of layers, number of quantizers, target audio lengths for each stage, etc. It is used to instantiate the model during training and inference. A "training config" contains hyperparameters for training the model. It is used to instantiate the trainer classes during training. See the `./configs` directory for example configs. ## Training ### CLAP RVQ The first step is to train the residual vector quantizer that maps continu
Excerpt of 11,221 characters
Read on GitHub
218
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:54e1aa4b70349f57, topic:music-generation, readme:music generation
matched fp:54e1aa4b70349f57, topic:transformer