Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Code for the paper "LLark: A Multimodal Instruction-Following Language Model for Music" by Josh Gardner, Simon Durand, Daniel Stoller, and Rachel Bittner.
| Date | Stars |
|---|---|
| 2026-07-31 | 383 |
| 2026-08-06 | 383 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# LLark: A Multimodal Instruction-Following Language Model for Music
This is the code release associated with the paper:
> LLark: A Multimodal Instruction-Following Language Model for Music<br />
> Josh Gardner, Simon Durand, Daniel Stoller, Rachel Bittner.<br />
> Published at ICML 2024
This repository contains the code used to build the training dataset, preprocess existing open-source music datasets, train the model, and run inference. **Note that this paper is not accompanied with any trained models.**
For more details about this work, you can read the [preprint of the paper](https://arxiv.org/abs/2310.07160), read the [Spotify Research blog post](https://research.atspotify.com/2023/10/llark-a-multimodal-foundation-model-for-music/) about this work, and [listen to
demos on the companion site](https://storage.googleapis.com/music2text-public/index.html).
See also the [ICML page for the paper](https://icml.cc/virtual/2024/poster/34440).
**This is not an officially supported Spotify product.**
## Citing This Repository
If you use this code in a publication, please cite our paper:
```
@article{gardner2023llark,
title={LLark: A Multimodal Instruction-Following Language Model for Music},
author={Gardner, Josh and Durand, Simon and Stoller, Daniel and Bittner, Rachel},
journal={Proc. of the International Conference on Machine Learning (ICML)},
year={2024}
}
```
# Using The Code
## Docker Environments
All of the code in this repo should be run in one of the provided Docker containers (see `docker`). There are three Dockerfiles for separate use cases:
* `m2t-train.dockerfile` is for model training and inference
* `m2t-preprocess.dockerfile` is for data preprocessing (i.e. running Beam pipelines)
* `jukebox-embed.dockerfile` is for extracting Jukebox embeddings
> **Warning**
>
> Note that the data preprocessing pipeline can be run locally or on [Google Cloud Dataflow](https://cloud.google.com/dataflow), a cloud-based data processing system that allows the code to be sped up by running in parallel across potentially thousands of machines. **Using these pipelines on Dataflow may incur costs against your own Google Cloud project.**
>
> Running these pipelines on Google Cloud Dataflow may also require pushing one or more Docker images to [Google Artifact Registry](https://cloud.google.com/artifact-registry), properly setting up permissions to use these Docker images from within Google Cloud Dataflow, and replacing various hard-coded Docker image paths with paths to your uploaded Docker images. **This repository contains no code to aid with this process.**
## Preprocess Your Own Audio
We provide some utilitities for converting, cropping, and annotating audio datasets using the process described in our paper. These utilities are located in `scripts/preprocessing`; see the header of each file in that directory for usage examples.
All of the preprocessing scripts use Apache Beam (on Google DataFlow). In order to run these code examples, we recommend using the Docker environment defined in `docker/m2t-preprocess.dockerfile`.
## Generate Instruction-Tuning Data
The file `scripts/openai/fetch_openai_instruct_data.py` can be adapted to generate instruction-tuning data. The prompts for each dataset are located in the `m2t/instruct` directory.
## Extract Jukebox Embeddings
The Jukebox embedding pipeline uses Apache Beam (on Google Cloud Dataflow). In order to run these code examples, we recommend using the Docker environment defined in `docker/jukebox-embed.dockerfile`. Embedding a set of around 100k audio files takes less than 1 hour using the default parameters in that script.
For using CLAP, we provide a similar set of utilities in the `scripts/clap` subdirectory; the CLAP embedding script can be executed from the `m2t-preprocess.dockerfile` Docker environment and does not require its own separate Docker environment.
## Training
This reExcerpt of 5,959 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:db165f235643cd66, topic:multimodal, desc:multimodal