Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Codebase for Aria - an Open Multimodal Native MoE
| Date | Stars |
|---|---|
| 2026-07-24 | 1086 |
| 2026-07-25 | 1086 |
| 2026-07-28 | 1086 |
| 2026-07-30 | 1086 |
| 2026-08-06 | 1086 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Aria
😊 [Hugging Face](https://huggingface.co/rhymes-ai/Aria) |
📄 [Paper](https://arxiv.org/pdf/2410.05993) |
📚 [Blog](https://rhymes.ai/blog-details/aria-first-open-multimodal-native-moe-model) |
🌐 [WebDemo](https://rhymes.ai/) |
🟣 [Discord](https://discord.com/invite/u8HxU23myj)
## Introduction
Aria is a multimodal native MoE model. It features:
- State-of-the-art performance on various multimodal and language tasks, superior in video and document understanding;
- Long multimodal context window of 64K tokens;
- 3.9B activated parameters per token, enabling fast inference speed and low fine-tuning cost.
## News
- [Jan 20, 2025] 🚀🚀🚀 Aria is supported in [PaddleMIX](https://github.com/PaddlePaddle/PaddleMIX/tree/develop/paddlemix/examples/aria) by Paddle Team.
- [Dec 15, 2024] We release [Aria-Chat](https://huggingface.co/rhymes-ai/Aria-Chat)! It is optimized for open-ended and multi-round dialogs, with enhanced reliability and multi-lingual support.
- [Dec 1, 2024] We release the base models for Aria ([Aria-Base-8K](https://huggingface.co/rhymes-ai/Aria-Base-8K) and [Aria-Base-64K](https://huggingface.co/rhymes-ai/Aria-Base-64K))! They are fully compatible with this inference \& fine-tuning codebase.
- [Oct 10, 2024] We release Aria!
## Quick Start
### Installation
```bash
pip install -e .
# or install with dev dependencies if you want to contribute to the project
pip install -e .[dev]
pip install grouped_gemm
pip install flash-attn --no-build-isolation
```
### Inference
Aria has 25.3B total parameters, it can be loaded in one A100 (80GB) GPU with bfloat16 precision.
Here is a code snippet to show you how to use Aria with Hugging Face Transformers.
```python
import requests
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor
model_id_or_path = "rhymes-ai/Aria"
model = AutoModelForCausalLM.from_pretrained(model_id_or_path, device_map="auto", torch_dtype=torch.bfloat16, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(model_id_or_path, trust_remote_code=True)
image_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png"
image = Image.open(requests.get(image_path, stream=True).raw)
messages = [
{
"role": "user",
"content": [
{"text": None, "type": "image"},
{"text": "what is the image?", "type": "text"},
],
}
]
text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=text, images=image, return_tensors="pt")
inputs["pixel_values"] = inputs["pixel_values"].to(model.dtype)
inputs = {k: v.to(model.device) for k, v in inputs.items()}
with torch.inference_mode(), torch.cuda.amp.autocast(dtype=torch.bfloat16):
output = model.generate(
**inputs,
max_new_tokens=500,
stop_strings=["<|im_end|>"],
tokenizer=processor.tokenizer,
do_sample=True,
temperature=0.9,
)
output_ids = output[0][inputs["input_ids"].shape[1]:]
result = processor.decode(output_ids, skip_special_tokens=True)
print(result)
```
We offer additional inference methods, such as utilizing [vLLM](https://github.com/vllm-project/vllm) for enhanced performance. For comprehensive details, please refer to [docs/inference.md](docs/inference.md).
### Cookbook
Checkout these [inference examples](https://github.com/rhymes-ai/Aria/tree/main/inference/notebooks) that demonstrate how to use Aria on various applications such as chart understanding, PDF reading, video understanding, etc, available with both Hugging Face Transformers and [vLLM](https://github.com/vllm-project/vllm) backends.
## Fine-tuning
> ⚠️ **Important Note on Fine-tuning**: Due to changes in the weight mapping after Aria's integration into transformers, the training code requires specific versions to work properly:
> - Use transformers version 4.45.0
> - Use model revision "4844f0b5ff678e768236889df5accbe4967ec845"
Excerpt of 8,439 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:d91f93b95db2c5d1, topic:multimodal, desc:multimodal, readme:multimodal
