Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A library for easily merging multiple LLM experts, and efficiently train the merged LLM.
| Date | Stars |
|---|---|
| 2026-07-31 | 517 |
| 2026-08-04 | 517 |
| 2026-08-06 | 517 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<h1>Mergoo
<img alt='Leeroo logo' src='https://github.com/Leeroo-AI/mergoo/blob/main/static/logo.png?raw=true' width='148' align='right' />
</h1>
[](#python)
[](https://www.gnu.org/licenses/lgpl-3.0.en.html)
[](https://pypi.org/project/mergoo/)
`mergoo` is a library for easily merging multiple LLM experts, and efficiently train the merged LLM. With `mergoo`, you can efficiently integrate the knowledge of different generic or domain-based LLM experts.
<img src='https://github.com/Leeroo-AI/mergoo/blob/main/static/base_light.png?raw=true' />
## 🚀 Features
- Supports several merging methods: **Mixture-of-Experts**, **Mixture-of-Adapters**, and **Layer-wise merging**
- Flexible merging for each layer
- Base Models supported : [Llama](https://llama.meta.com/)(including LLaMa3), [Mistral](https://huggingface.co/docs/transformers/en/model_doc/mistral), [Phi3](https://huggingface.co/docs/transformers/main/en/model_doc/phi3), and [BERT](https://huggingface.co/docs/transformers/en/model_doc/bert)
- Trainers supported : 🤗 [Trainer](https://huggingface.co/docs/transformers/en/main_classes/trainer), [SFTrainer](https://huggingface.co/docs/trl/en/sft_trainer), [PEFT](https://huggingface.co/docs/peft/en/index)
- Device Supported: CPU, MPS, GPU
- Training choices: Only Router of MoE layers, or Fully fine-tuning of Merged LLM
If you like the project, consider leaving a ⭐️
## Installation
Install by pip:
```
pip install mergoo
```
Install latest unstable version on Github:
```
pip install git+https://github.com/Leeroo-AI/mergoo
```
Install it from the source:
```
git clone https://github.com/Leeroo-AI/mergoo
cd mergoo
pip install -e .
```
## Quick Start
### Configuration Setup
Specify the config for merging:
- ```model_type```: type of base model. choices: ```mistral```, ```llama```, or ```bert```.
- ```num_experts_per_token```: Number of experts for each token of MoE.
- ```experts```: config for experts to merge. includes ```expert_name``` and Hugging Face 🤗```model_id```.
- ```router_layers```: layers chosen for applying Mixture-of-Experts.
#### Fully Fine-tuned Experts
This is a sample config when merging **fully** fine-tuned LLM experts.
```python
config = {
"model_type": "mistral",
"num_experts_per_tok": 2,
"experts": [
{"expert_name": "base_expert", "model_id": "mistralai/Mistral-7B-v0.1"},
{"expert_name": "expert_1", "model_id": "meta-math/MetaMath-Mistral-7B"},
{"expert_name": "expert_2", "model_id": "ajibawa-2023/Code-Mistral-7B"}
],
"router_layers": ["gate_proj", "up_proj", "down_proj"]
}
```
For the above example, we merged math and code mistral-based experts. Please refer to [this notebook](https://github.com/Leeroo-AI/mergoo/blob/main/notebooks/llama_compose_trainer.ipynb) for further details!
#### Mixture of Adapters (MoE on LoRA)
This is a sample config when merging **LoRA** fine-tuned LLM experts. ```mergoo``` builds a routing layer on top of LoRAs, resulting in a **mixture of adapters**.
```python
config = {
"model_type": "mistral",
"num_experts_per_tok": 2,
"base_model": "mistralai/Mistral-7B-v0.1",
"experts": [
{"expert_name": "adapter_1", "model_id": "predibase/customer_support"},
{"expert_name": "adapter_2", "model_id": "predibase/customer_support_accounts"},
{"expert_name": "adapter_3", "model_id": "predibase/customer_support_orders"},
{"expert_name": "adapter_4", "model_id": "predibase/customer_support_payments"}
],
}
```
The ```expert_name``` starts with ```adapter``` instead of ```expert```. Please refer to [this notebook](https://github.com/Leeroo-AI/mergoo/blob/main/notebooks/Mistral_lora_compose_trainer.ipynb) for further details!
### Merge Experts
Following the config setup, ```mergoo``Excerpt of 8,503 characters
Read on GitHub
30
6
2
2
1
Ikko Eltociear Ashimine · Japan
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:8671ca97093ad7dc, topic:fine-tuning, topic:lora
matched fp:8671ca97093ad7dc, topic:large-language-models, topic:llm
matched fp:8671ca97093ad7dc, topic:nlp