Upload a photo · Clone a voice · Talk to any face in real time
Quick Start · Features · Architecture · GPU / AWS Deploy · API · Roadmap
The most complete open-source AI avatar / digital human system. Real-time talking-head lip-sync · Zero-shot voice cloning · Multi-LLM · Runs 100% locally or on AWS.
AvatarAI is an open-source, production-ready platform for building photorealistic AI avatar conversations. Upload any face photo, clone a voice from a 5-second audio clip, and have a real-time conversation — with lip-sync video generated on every single response.
[mic] → Whisper STT → Claude / GPT / Ollama (streaming) → Chatterbox TTS → MuseTalk lip-sync → [video]
< 2–4 s to first video chunk on AWS GPU >
What makes AvatarAI different:
- 🎤 Zero-shot voice cloning — 10 seconds of audio is all you need (Chatterbox Multilingual)
- 🎭 Any face, any language — upload a JPEG, pick from 23 languages, start talking
- ⚡ Token-streaming pipeline — the LLM streams live tokens while TTS + lip-sync run per sentence; the first video chunk plays before the model finishes its reply
- ✋ Barge-in — speak (or hit stop) mid-reply and the avatar yields instantly, like a real conversation
- 🔒 100% local mode — local storage, local Whisper, local LLM via Ollama: nothing leaves your machine
- 🔌 Multi-LLM — Claude (with prompt caching), GPT-4o, or any local model via Ollama / vLLM / LM Studio
- 🚀 AWS GPU deployment — one-command deploy to
g5.xlargefor true real-time (~30 FPS) - 🏗️ Production-grade — JWT + httpOnly-cookie auth, per-user rate limiting, Postgres + Alembic, S3/CloudFront, Prometheus, CI, a real test suite — the only project in this niche you can ship as a product, not just a demo
| AvatarAI | Duix-Avatar | Linly-Talker | AIAvatarKit | |
|---|---|---|---|---|
| Real-time conversation | ✅ WebSocket streaming | ❌ offline video gen | ✅ (Gradio / WebRTC spin-off) | ✅ |
| Lip-sync video | ✅ MuseTalk V1.5 | ✅ proprietary models | ✅ multiple engines | ❌ (drives external avatars) |
| Voice cloning | ✅ 10 s, 23 languages | ✅ | ✅ | ❌ |
| Barge-in / interruption | ✅ | ❌ | ✅ (stream variant) | ✅ |
| Local / free LLM | ✅ Ollama, vLLM | ❌ | ✅ | ✅ |
| Web app with auth & history | ✅ Next.js + JWT + Postgres | ❌ Windows client | ❌ Gradio demo UI | ❌ library |
| Rate limiting, CI, tests, IaC | ✅ | ❌ | ❌ | ❌ |
| License | MIT | custom | MIT | Apache-2.0 |
Toolkits like Linly-Talker are great research playgrounds; Duix ships a Windows product. AvatarAI is the one you can deploy as a real multi-user web service.
| Category | Details |
|---|---|
| 🤖 LLM Backends | Claude (prompt-cached) · GPT-4o · Ollama / vLLM / LM Studio (local, free) |
| 🎤 Voice Cloning | Record 10–60 s → Chatterbox Multilingual zero-shot cloning |
| 🗣️ Speech-to-Text | Whisper (faster-whisper, CUDA), decodes browser WebM natively |
| 🎬 Lip-Sync Video | MuseTalk V1.5 persistent worker (30 FPS on GPU) · optional GFPGAN restore · FFmpeg fallback (CPU) |
| ⚡ Streaming Pipeline | Live LLM tokens + per-sentence video chunks over WebSocket |
| ✋ Barge-In | Speak or hit stop mid-reply — in-flight turn cancels in ms |
| 🔉 TTS Fallback Chain | chatterbox (optional, separate venv) → edge-tts (free neural voices) → gTTS — never silent |
| 😊 Emotion Detection | Live emotion badges per message |
| 🌍 23 Languages | Whisper multilingual STT + Chatterbox multilingual TTS |
| 🏠 Local-First Storage | USE_LOCAL_STORAGE=true — no AWS needed for dev |
| 🔐 Auth & Sessions | JWT authentication, conversation history, persistent sessions |
| 📊 Observability | Prometheus · Celery Flower · Sentry · structured logging |
| 🧪 Tested | Full pytest suite — users, avatars, sessions, health checks |
| 🚀 AWS GPU Deploy | One-command g5.xlarge deploy with CUDA 11.8 + float16 |
┌──────────────────────────────────────────────────────────────┐
│ Browser / Client │
│ ┌─────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
│ │Avatar Studio│ │ Voice Studio │ │ Chat Interface │ │
│ │ (upload) │ │ (cloning) │ │ Idle anim + chunks │ │
│ └──────┬──────┘ └──────┬───────┘ └──────────┬───────────┘ │
└─────────┼───────────────┼─────────────────────┼─────────────┘
│ REST │ REST │ WebSocket
▼ ▼ ▼
┌──────────────────────────────────────────────────────────────┐
│ FastAPI Backend │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ WebSocket Manager │ │
│ │ split sentences → TTS → MuseTalk → stream chunks │ │
│ └──────────────────────────────────────────────────────┘ │
│ ┌──────────┐ ┌───────────┐ ┌──────────┐ ┌───────────────┐ │
│ │ Whisper │ │Claude/GPT │ │ XTTS v2 │ │ MuseTalk │ │
│ │ STT │ │ / Llama │ │ TTS │ │ (GPU/CPU) │ │
│ └──────────┘ └───────────┘ └──────────┘ └───────────────┘ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐ │
│ │PostgreSQL│ │ Redis │ │ Celery │ │ Local FS / S3 │ │
│ └──────────┘ └──────────┘ └──────────┘ └───────────────┘ │
└──────────────────────────────────────────────────────────────┘
[User types / speaks]
│
▼
Whisper STT ─────────────────► transcript
│
▼
Claude / GPT / Llama ────────► full response text
│
▼
Split into sentences ────────► ["Hello!", "How are you?", ...]
│
├── sentence 1 → XTTS → MuseTalk → video_chunk WS → browser plays
├── sentence 2 → XTTS → MuseTalk → video_chunk WS → queued
└── sentence N → XTTS → MuseTalk → video_chunk WS → queued
ai-avatar-system/
├── backend/ # FastAPI application
│ ├── app/
│ │ ├── api/v1/ # REST endpoints (users, avatars, sessions, messages)
│ │ ├── services/ # Core services (LLM, TTS, STT, animator, storage)
│ │ ├── models/ # SQLAlchemy DB models
│ │ └── websocket.py # Real-time WebSocket handler + sentence streaming
│ ├── alembic/ # Database migrations
│ ├── models/MuseTalk/ # MuseTalk V1.5 (lip-sync engine)
│ │ └── scripts/
│ │ └── musetalk_worker.py # Persistent worker (models loaded once)
│ ├── tests/ # pytest suite
│ ├── Dockerfile # CUDA 11.8 base image
│ └── requirements.txt
├── frontend/ # Next.js 16 application
│ ├── app/ # App Router pages
│ ├── components/ # React components (ChatInterface, IdleAvatar, etc.)
│ ├── lib/api.ts # Axios API client
│ └── store/ # Zustand global state
├── nginx/
│ └── nginx.conf # Reverse proxy (HTTP → backend/frontend, WebSocket)
├── infrastructure/
│ ├── main.tf # AWS Terraform (ECS, RDS, ElastiCache, S3, CloudFront)
│ └── variables.tf
├── scripts/
│ ├── setup_musetalk.sh # Download MuseTalk models (~9 GB)
│ └── deploy-aws.sh # One-command EC2 GPU deployment
├── docker-compose.yml # Development (CPU) — all services
├── docker-compose.prod.yml # Production overrides (GPU, no bind mounts, logging)
├── deploy.sh # ECR push + Terraform deploy (ECS path)
├── .env.example # Development env template
└── .env.prod.example # Production env template
- Docker & Docker Compose v2+ (recommended)
- OR: Python 3.10+, Node.js 18+, FFmpeg, PostgreSQL, Redis
git clone https://github.com/PunithVT/ai-avatar-system.git
cd ai-avatar-system
cp .env.example .env # add your ANTHROPIC_API_KEY (or OPENAI_API_KEY)
docker compose up -d| Service | URL |
|---|---|
| 🖥️ Frontend | http://localhost:3000 |
| ⚙️ Backend API | http://localhost:8000 |
| 📖 Swagger Docs | http://localhost:8000/docs |
| 🌸 Celery Flower | http://localhost:5555 |
No AWS required. Set
USE_LOCAL_STORAGE=true(default) — uploads saved tobackend/uploads/.
Want something to talk to immediately? Seed three ready-made demo avatars (AI-generated faces + personalities):
backend/venv/bin/python scripts/seed_demo.py # or any python with `requests`
backend/venv/bin/python scripts/seed_demo.py --with-voices # + cloned demo voicesPrebuilt images are also published on every release — ghcr.io/punithvt/ai-avatar-system-backend and …-frontend.
# Backend
cd backend
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
cp ../.env.example ../.env
alembic upgrade head
uvicorn main:app --reload --port 8000
# Frontend (new terminal)
cd frontend
npm install
npm run dev# Download models (~9 GB, one-time)
bash scripts/setup_musetalk.sh
# Set in .env
AVATAR_ENGINE=musetalk
# Restart
docker compose restart backendMuseTalk achieves 30 FPS at 256×256 on a V100-class GPU (source: MuseTalk paper). On CPU it is 30–50× slower. Deploying on AWS gets you genuine real-time performance.
| Instance | GPU | VRAM | Spot $/hr | MuseTalk FPS |
|---|---|---|---|---|
g4dn.xlarge |
T4 | 16 GB | ~$0.16 | ~15–20 FPS |
g5.xlarge |
A10G | 24 GB | ~$0.30 | ~30 FPS ✓ |
g6.xlarge |
L4 | 24 GB | ~$0.24 | ~30 FPS ✓ |
Recommended: g5.xlarge Spot (~$72/mo at 8 hrs/day).
# 1. Launch g5.xlarge with Ubuntu 22.04 LTS, SSH in, then:
bash <(curl -fsSL https://raw.githubusercontent.com/PunithVT/ai-avatar-system/main/scripts/deploy-aws.sh)
# 2. Fill in API keys:
nano /opt/ai-avatar-system/.env.prod
# 3. Redeploy with your keys:
bash /opt/ai-avatar-system/scripts/deploy-aws.sh --updateThe script automatically:
- Installs Docker + nvidia-docker2
- Verifies GPU is accessible
- Downloads MuseTalk models (~9 GB)
- Starts all services with GPU passthrough + float16 (2× faster via Tensor Cores)
cp .env.prod.example .env.prod # fill in your values
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -dWhat docker-compose.prod.yml adds over development:
- GPU reservation (
nvidiadriver, count=1) for backend + celery-worker float16inference enabled automatically on CUDA → ~2× speedup- Persistent
musetalk_modelsvolume (survive container restarts) - No source-code bind mounts (runs from built image)
- Log rotation (100 MB max, 5 files)
- Flower disabled (security)
# Check GPU is visible in container
docker exec avatar-backend python -c "
import torch
print('CUDA:', torch.cuda.is_available())
print('GPU:', torch.cuda.get_device_name(0))
print('VRAM:', round(torch.cuda.get_device_properties(0).total_memory/1024**3,1), 'GB')
"
# Expected on g5.xlarge:
# CUDA: True
# GPU: NVIDIA A10G
# VRAM: 24.0 GB
# Live GPU utilisation
docker exec avatar-backend nvidia-smiFor a fully managed ECS deployment with RDS + ElastiCache + CloudFront:
cd infrastructure
terraform init
terraform apply -var="environment=production"
bash deploy.sh productionAVATAR_ENGINE selects how video is generated. MuseTalk is the default and
stays installed whatever else you add; any engine failure falls back to
simple rather than failing the turn.
| Engine | VRAM | Speed | Use it for |
|---|---|---|---|
simple |
none | instant | CPU hosts, no lip-sync |
musetalk |
16–24 GB | ~30 FPS | default — live conversation |
liveavatar |
48 GB (FP8) / 80 GB | minutes per turn | offline renders where fidelity beats latency |
Alibaba LiveAvatar (Wan2.2-S2V-14B
- LoRA, Apache 2.0) produces higher fidelity than MuseTalk and supports 10,000+ second continuous generation from a single reference image.
bash scripts/setup_liveavatar.sh # checks VRAM first, then ~60 GB of weights
# then in .env:
AVATAR_ENGINE=liveavatarIt is not a MuseTalk replacement, and MuseTalk remains the default. Two reasons, both structural:
- 48 GB VRAM minimum with FP8, 80 GB without. That is a different hardware
tier from the
g5.xlargethis project targets. - It loads the 14B model on every invocation. Upstream's entry point
(
minimal_inference/s2v_streaming_interact.py) runs once and exits — there is no persistent-worker mode, which is exactly what MuseTalk uses to keep per-turn cost to inference alone. So a turn costs minutes, not milliseconds.
That makes LiveAvatar a fit for the offline Celery render path, where a
slow better-looking result is the point, and a poor fit for live conversation.
Switching back is one setting: AVATAR_ENGINE=musetalk.
Giving LiveAvatar a persistent worker would mean patching upstream. That is the maintenance debt that got SadTalker removed from this project, so it has deliberately not been done.
MuseTalk regenerates the mouth region at 256×256 and the worker scales it
back up to the crop size, while the avatar itself is stored at
AVATAR_RESOLUTION (512). The mouth therefore renders at roughly half the
resolution of the face around it — and that gap, not the model, is the
pipeline's visible quality ceiling. Swapping lip-sync engines doesn't close it;
a face restorer does.
bash scripts/setup_face_restore.sh # installs gfpgan + ~520 MB of weights
# then in .env:
FACE_RESTORE=gfpgan # off | gfpgangfpgan is deliberately not in requirements.txt. It depends on
basicsr, which ships no wheel and builds from source, and whose setup.py
collides with the CUDA base image's preinstalled cuda-toolkit — listing it
there failed the backend image build for everyone, including people who never
enable restoration. It lives in backend/requirements-face-restore.txt and is
installed by the setup script, the same way MuseTalk and Chatterbox are.
For the same reason, restoration is a bare-metal / GPU-host feature today:
inside the CUDA container the basicsr build needs work that hasn't been done
yet.
GFPGAN runs over each composited frame after MuseTalk has blended its generated mouth back in, so it sees a complete face to detect.
It is off by default, deliberately. Restoration runs per frame, so it
trades FPS for sharpness, and whether that trade is worth it depends on your
GPU and how close to real-time you need to stay. A/B it against
FACE_RESTORE=off on the same avatar and audio before leaving it on.
Everything about it degrades rather than breaks: missing weights, a broken
gfpgan install, CPU-only hardware, or a single frame that fails to restore
all fall back to plain MuseTalk output. Lip-sync is the feature; sharpness is
the nicety.
The LLM context is a fixed window of the most recent turns (MAX_CONTEXT_MESSAGES,
60). Anything older used to be dropped outright, so a long conversation quietly
forgot its own beginning — the avatar would lose your name twenty minutes in.
Turns leaving the window are now folded into a running summary carried in the
system prompt, and persisted to Conversation.summary so a browser refresh
doesn't lose it.
CONVERSATION_MEMORY=true # default
MEMORY_SUMMARY_MAX_CHARS=1500A summary rather than embeddings, deliberately. It costs one LLM call when the window overflows — roughly once every 30 turns, not once per reply — needs no vector store or embedding model, and works with every provider the project supports. Requiring an embeddings API would have broken the "runs fully local and free with Ollama" path, since Anthropic has no embeddings endpoint and a local embedding model is another dependency to install and keep working.
The tradeoff is honest: a summary is lossy. It carries the thread of a conversation, not verbatim recall of a detail from an hour ago. Fact-level retrieval is what RAG would add, and that remains open on the roadmap.
Failure never costs a turn — if summarisation fails, the window truncates exactly as it did before and the reply proceeds.
Tap the Radio button next to the mic and just talk — no tap-to-record. The avatar detects when you start and stop speaking, and speaking over a reply interrupts it.
How turns are decided (frontend/lib/vad.ts):
- Adaptive threshold. The detector measures the room's ambient level for ~700 ms, then listens for speech a margin above it, and keeps tracking the floor while nobody is talking. A fixed threshold works at a desk and fails in a cafe.
- Pause tolerance. A turn ends after ~900 ms of continuous silence, so "I think… maybe we should" stays one turn instead of three.
- Noise rejection. Anything shorter than ~300 ms is discarded rather than sent, so a cough or a door doesn't cost an STT call and an LLM turn.
- Echo cancellation is requested on the mic. Without it the open mic hears the avatar through the speakers and the session talks to itself.
The detector is pure — it takes (level, timestamp) samples and returns turn
boundaries — so its timing logic can be exercised against a synthetic trace
without a browser or a microphone.
Powered by Chatterbox Multilingual (Resemble AI) — zero-shot voice cloning from a 10-second sample, in 23 languages.
- Go to Voice tab → Clone Voice
- Record 10–60 s of clear speech (or upload a WAV/MP3/WebM)
- Name it → Clone → select it for your session
Every TTS response then uses your cloned voice.
Chatterbox needs its own environment. It is not in
requirements.txt, because it pinstorch==2.6.0/transformers==5.2.0/librosa==0.11.0while the MuseTalk lip-sync pipeline needs torch 2.2 / transformers 4.37 — the two cannot share one virtualenv. Install it separately:python -m venv venv-tts ./venv-tts/bin/pip install -r backend/requirements-tts.txtWithout it nothing breaks: TTS falls through to edge-tts (free Microsoft neural voices, CPU-only) and then gTTS. Voice cloning is the only feature that requires Chatterbox, so set
TTS_PROVIDER=chatterboxonly once it is actually installed.
# REST API
curl -X POST http://localhost:8000/api/v1/voices/clone \
-F "audio=@my_voice.wav" -F "name=My Voice" -F "language=en"POST /api/v1/users/register { "email": "...", "username": "...", "password": "..." }
POST /api/v1/users/login form: username=... password=... → { "access_token": "..." }
POST /api/v1/users/guest (no body) → { "access_token": "..." }
POST /api/v1/users/logout clears the httpOnly auth cookie
# Every resource route requires an identity — pass the bearer token:
Authorization: Bearer <access_token>
# ...or rely on the httpOnly cookie that /login and /guest set for browsers./guest backs the UI's Continue as Guest button. It creates a real but
anonymous account, so a guest's avatars and transcripts are scoped to them
alone — guests never share data with each other. Guest accounts have no
usable password (they are reachable only via the token above) and are deleted
after GUEST_RETENTION_HOURS of inactivity by the cleanup_guest_accounts
task. Set GUEST_ACCOUNTS_ENABLED=false to require sign-up.
Registration rules: passwords are 8–128 characters, usernames are 3–39
characters of [A-Za-z0-9_-] (must start and end alphanumeric), and email
addresses are normalised to lower case so [email protected] and [email protected] are
the same account.
POST /api/v1/avatars/upload Upload photo (multipart: file + name)
limits: MAX_UPLOAD_SIZE (413 if over),
ALLOWED_EXTENSIONS, name ≤ 200 chars
GET /api/v1/avatars/ List avatars
DELETE /api/v1/avatars/{id} Delete avatar
PUT /api/v1/avatars/{id}/voice Assign voice to avatar
POST /api/v1/sessions/create { "avatar_id": "..." }
POST /api/v1/sessions/{id}/end
GET /api/v1/messages/session/{id}
WS /ws/session/{session_id}
Client → Server:
{ "type": "text", "text": "Hello!" }
{ "type": "audio", "audio": "<base64-webm>" }
{ "type": "stop" } // barge-in: cancel the in-flight reply
{ "type": "set_voice", "voice_id": "<uuid>" } // attach a cloned voice (owner-checked)
{ "type": "set_language", "language": "es" }
{ "type": "ping" }Server → Client:
{ "type": "token", "token": "Hel" } // live LLM stream
{ "type": "transcription", "text": "Hello!" }
{ "type": "message", "content": "Hi!", "role": "assistant" }
{ "type": "video_chunk_start","total_chunks": -1 } // -1 = streaming, total unknown
{ "type": "video_chunk", "chunk_index": 0, "video_url": "...", "text": "Hi!" }
{ "type": "video_chunk_end", "sent_chunks": 3 }
{ "type": "status", "message": "Animating…", "stage": "animation" }
{ "type": "tts_fallback", "engine": "edge-tts", "voice_cloned": false, "message": "…" }
{ "type": "interrupted", "message": "Previous response interrupted" }
{ "type": "error", "message": "Something went wrong" }Key .env variables:
# LLM
LLM_PROVIDER=anthropic # anthropic | openai | ollama (local & free)
LLM_MODEL=claude-sonnet-4-6 # or gpt-4o · llama3.1 · qwen2.5 …
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_BASE_URL= # e.g. http://localhost:11434/v1 for Ollama / vLLM / LM Studio
# Avatar engine
AVATAR_ENGINE=musetalk # musetalk (default) | liveavatar (48GB+) | simple (CPU)
MUSETALK_PATH=models/MuseTalk
# TTS — automatic fallback chain: chatterbox → edge-tts → gtts
# chatterbox requires backend/requirements-tts.txt in its own venv (see
# "Voice Cloning"); without it this transparently uses edge-tts.
TTS_PROVIDER=chatterbox
# STT
WHISPER_MODEL=large-v3-turbo # tiny | base | small | medium | large-v3 | large-v3-turbo
# Storage
USE_LOCAL_STORAGE=true # false → AWS S3 (+ presigned URLs / CloudFront)
S3_BUCKET_NAME=...
# Auth (≥32 chars enforced at boot)
SECRET_KEY=$(python -c "import secrets; print(secrets.token_hex(32))")
JWT_SECRET_KEY=$(python -c "import secrets; print(secrets.token_hex(32))")
JWT_EXPIRATION_HOURS=24| Library | Purpose |
|---|---|
| Next.js 16 + React 18 | App framework |
| TypeScript 5 | Type safety |
| Tailwind CSS | Styling |
| Zustand | Global state |
| Library | Purpose |
|---|---|
| FastAPI | Async REST API + WebSocket |
| SQLAlchemy 2 (async) | ORM with asyncpg |
| PostgreSQL 15 | Primary database |
| Alembic | Migrations |
| Redis 7 | Cache + Celery broker |
| Celery | Background tasks |
| Model | Purpose |
|---|---|
| Claude / GPT-4o / Ollama (local) | LLM conversation |
Whisper (faster-whisper) |
Speech-to-text |
| Chatterbox Multilingual (Resemble AI) | TTS + zero-shot voice cloning, 23 languages |
| Edge TTS → gTTS | Free no-GPU fallback voices |
| MuseTalk V1.5 | Photorealistic lip-sync (30 FPS on GPU) |
cd backend
pytest -v # all tests
pytest tests/test_health.py # single module
pytest --cov=app --cov-report=html # HTML coverage- 2026-06 — Edge-TTS neural fallback chain · local LLMs via Ollama/vLLM · demo avatar seeding · prebuilt GHCR images · cascade-delete + WebM-STT + 429 fixes · SEO/metadata pass
- 2026-05 — httpOnly-cookie auth (XSS-safe) · conversation resume from history · end-to-end WebSocket tests · perf indexes (migration 0002)
- 2026-03 — Chatterbox Multilingual replaces XTTS v2 (23 languages) · MuseTalk persistent worker (models load once) · barge-in interruption · live token streaming
- Streaming LLM — TTS + lip-sync start before the LLM finishes (token-by-token) ✅
- Barge-in — interrupt the avatar mid-reply by speaking ✅
- Local LLMs — Ollama / vLLM / LM Studio via OpenAI-compatible API ✅
- Hands-free mode — VAD-driven always-listening with auto end-of-turn (no tap-to-record) ✅
- WebRTC streaming — sub-second full-duplex audio/video instead of chunked MP4
- Wav2Lip engine — lighter lip-sync option for weaker GPUs
- Emotion-driven animation — detected emotion changes facial expression
- Embeddable widget — drop a talking avatar into any website with 3 lines of JS
- Long-term memory — rolling summary carries context past the window ✅ (RAG + vector DB still open for fact-level recall)
- UI i18n — the pipeline speaks 23 languages; the UI should too
Do I need a GPU?
No — everything runs on CPU. MuseTalk takes 30–90 s/sentence on CPU (the simple engine is instant). For real-time lip-sync, use an AWS g5.xlarge (~$0.30/hr spot) or any 16 GB+ NVIDIA card.
Can I run it with no API key, fully offline?
Yes — set LLM_PROVIDER=ollama, run Ollama (ollama run llama3.1), and you have a fully local, free conversation stack: Whisper STT, local LLM, Chatterbox TTS, MuseTalk video.
How do I get something to talk to quickly?
Run python scripts/seed_demo.py — it creates three demo avatars (AI-generated faces, distinct personalities) and optionally cloned demo voices with --with-voices.
How do I get MuseTalk models?
Run bash scripts/setup_musetalk.sh — downloads ~9 GB of models automatically.
Why does the first response take longer?
The MuseTalk persistent worker loads all models into GPU VRAM on the first request (~60 s on GPU, ~5 min on CPU). Subsequent requests reuse the loaded models.
What happens if the TTS model can't load?
The pipeline degrades gracefully: chatterbox → edge-tts (free Microsoft neural voices) → gTTS. The UI shows a one-time notice when a cloned voice couldn't be applied.
What avatar photo works best?
A clear, well-lit frontal face photo (JPEG/PNG/WebP). Avoid sunglasses or heavy occlusion.
Contributions welcome! Read CONTRIBUTING.md before opening a PR.
git clone https://github.com/PunithVT/ai-avatar-system.git
git checkout -b feat/my-feature
# make changes + tests
git commit -m "feat(backend): add my feature"
git push origin feat/my-featureAvatarAI is MIT-licensed and developed in the open. If it saves you time, or your company is running it in production, sponsoring directly funds the work that is expensive to do for free:
- GPU time for testing the MuseTalk lip-sync pipeline across resolutions and hardware
- Model evaluation as new TTS and STT releases land, so the defaults stay current instead of drifting
- Maintenance — dependency and security upkeep, issue triage, and review
Not in a position to sponsor? Starring the repo, filing a good bug report, or writing about what you built with it all genuinely help.
MIT © 2026 — see LICENSE for details.
If AvatarAI saves you time or inspires your project, please ⭐ star the repo.
Built with FastAPI · Next.js · MuseTalk V1.5 · Chatterbox · Whisper · Claude AI