Working against the new Google Flow API. Flow moved to
flow.google.comin September 2026 and stopped minting theBearer ya29.…that the oldaisandbox-pa.googleapis.comREST API needed. Thebatchexecutetransport that replaces it is in place and verified end to end against the live API — image generation, 2K image export, and image-to-video all run green. Upgrading from an older Flow Kit: reload the extension (v0.3.2+) and pinFLOW_PROJECT_ID; Flow Kit can no longer create the project for you.Every Omni 1.1 Flash video mode is ported and live-verified: text-to-video, first frame, first+last frame, and references. Three capabilities remain unported on the Veo path because their payloads were never captured off the new UI — video upscale, Veo reference-to-video, and Veo start+end-frame chaining. They fail loudly with
UNSUPPORTED_ON_BATCH_APIinstead of quietly producing the wrong thing. For the latter two, Omni covers the same shot withmodel_family=omni_flash, orFLOW_ALLOW_DEGRADED=1drops them to plain i2v; video upscale has no fallback. To restore one properly seedocs/CAPTURE.md.
Standalone system to generate AI videos via Google Flow. Uses a Chrome extension as a browser bridge: it mints reCAPTCHA and runs Flow's batchexecute RPCs inside a signed-in flow.google.com tab, which is the only place they can be signed.
All outputs below were generated end-to-end by this system — from story concept to final YouTube-ready video with thumbnails, narration, and branding.
The reference image system keeps characters consistent across an entire video. Each character is generated once as a reference, then the AI uses that reference in every scene — maintaining the same face, clothing, and features.
Doctor character — same face, glasses, white coat across 4 different scenes:
Defector character — same face across ICU, hospital, interview, and Seoul streets:
All frames from a single 50-scene project. Both characters maintain consistent appearance across completely different settings and lighting conditions — powered by the reference image system.
Strategic briefing → pilot departure → formation flight → aircraft hit → CSAR alert → pilot survival.
Each project goes through: story → entities → reference images → scene images → 8s video clips → narration (TTS) → concat → thumbnails → YouTube upload — all orchestrated via API or AI agent skills.
| Output | Description |
|---|---|
| Reference images | One per character/location/prop — maintains visual consistency |
| Scene images | Composed using all referenced entities |
| 8-second video clips | Generated from scene images with camera motion + sound effects |
| 4K upscale | Optional upscale to 4K resolution |
| Narrator TTS | Voice-cloned narration per scene |
| Final video | All clips concatenated, trimmed to narrator timing |
| Thumbnails | YouTube-optimized with text overlays + branding |
| YouTube metadata | SEO-optimized title, description, tags, hashtags |
The Chrome extension runs alongside Google Flow — showing real-time request log (614 total, 328 success), video generation progress, and token status. The Python agent communicates with the extension via WebSocket to automate all API calls.
A local React dashboard (dashboard/) for monitoring and driving the pipeline — real-time KPIs, per-video stage progress, a scene-level pipeline view with AI review, and a setup guide, all backed by the same FastAPI agent. Supports English, Vietnamese, Hindi, Indonesian, Chinese, Korean, and Japanese.
┌──────────────────┐ WebSocket ┌──────────────────────┐ ┌──────────────────┐
│ Python Agent │◄──────────────────►│ Chrome Extension │────►│ flow.google.com │
│ (FastAPI+SQLite)│ localhost:9222 │ (MV3 Service Worker) │ │ (signed-in tab) │
│ │ │ │ │ │
│ - REST API :8100│ ── envelopes ──► │ - reCAPTCHA mint │ │ batchexecute │
│ - Queue worker │ ◄── responses ── │ - runs the RPC in │ │ cookie + `at` │
│ - Post-process │ │ the page's world │ │ │
│ - SQLite DB │ │ │ │ │
└──────────────────┘ └──────────────────────┘ └──────────────────┘
Flow signs every call with the session cookie plus a per-page at token, and a
generate also carries a single-use reCAPTCHA. None of that can be replayed from
outside the browser, so the agent builds the request and the page issues it.
One signed-in Flow tab has to stay open; nothing here works headless.
September 2026 — Flow moved. It now lives at
flow.google.comand the oldaisandbox-pa.googleapis.comREST API has no caller: theBearer ya29.…it needed stopped being minted. If you are upgrading from an older Flow Kit, reload the extension (v0.3.2+) and pinFLOW_PROJECT_ID— see Configuration. The REST path has been removed;git loghas it if a payload is ever needed for reference.
./setup.shThis checks and installs: Python 3.10+, pip, ffmpeg, ffprobe, Chrome, creates venv, installs dependencies, verifies imports.
Windows: Use WSL (
wsl --install) or Git Bash. All bash scripts and commands assume a Unix shell.
# Prerequisites: Python 3.10+, ffmpeg, Chrome
pip install -r requirements.txt# 1. Load Chrome extension: chrome://extensions → Developer mode → Load unpacked → extension/
# 2. Open https://flow.google.com/ and sign in — leave the tab open
# 3. Create a project in the Flow UI and copy its uuid out of the URL
export FLOW_PROJECT_ID=<that uuid>
# 4. Start agent
source venv/bin/activate # if using setup.sh
python -m agent.main
# 5. Verify
curl http://127.0.0.1:8100/health
# {"status":"ok","extension_connected":true}
curl http://127.0.0.1:8100/api/flow/status
# {"connected":true,"transport":"batch","flow_project_id":"…","flow_key_present":false}flow_key_present: false is expected — the current transport has no bearer
token. Step 3 is not optional: Flow's project-creation endpoint went with the
migration, so without a pinned project every request fails NO_FLOW_PROJECT.
You can also pass flow_project_id per project on POST /api/projects.
| Env var | Default | What it does |
|---|---|---|
FLOW_PROJECT_ID |
— | The Flow project every RPC is scoped to. Required. |
FLOW_ALLOW_DEGRADED |
0 |
1 lets scene chaining and r2v fall back to plain i2v instead of failing. |
DEFAULT_PAYGATE_TIER |
PAYGATE_TIER_TWO |
Carried for the DB and dashboard; no longer selects a model. |
The migrated image path supports Nano Banana Pro, Nano Banana 2 and Nano Banana
2 Lite, all five current aspect ratios, 1-4 outputs, true base-image editing and
native 2K image export. Exact future Flow image model wire ids pass through
without being silently replaced by the default model. See
docs/IMAGE_API.md.
Three capabilities have no captured payload, so they fail with
UNSUPPORTED_ON_BATCH_API rather than quietly producing the wrong thing:
| Capability | Status | Workaround |
|---|---|---|
4K/1080p upscale (/fk-pipeline last step) |
unported | none — keep the 1080p render |
| Veo reference-to-video (r2v) | unported | Omni r2v (model_family=omni_flash), or FLOW_ALLOW_DEGRADED=1 → i2v off the first reference |
Veo start+end-frame chaining (/fk-gen-chain-videos) |
unported | Omni first+last (model_family=omni_flash), or FLOW_ALLOW_DEGRADED=1 → i2v off the start frame |
| Omni Flash text-to-video | ported | POST /api/flow/generate-video-omni-text (4/6/8/10s) |
| Omni Flash frame / first+last / reference modes | ported | eb1hJf, nprQif, MZZa6b — POST /api/flow/generate-video with model_family=omni_flash |
Restoring one starts with a capture, not a guess: docs/CAPTURE.md.
A chubby cat sells fish at a market. 3 scenes, vertical, Pixar 3D style.
The system uses reference images to keep visuals consistent across scenes. Here's the mental model:
1. Identify every visual element that should look the same across scenes:
- Characters →
entity_type: "character"(portrait reference) - Places →
entity_type: "location"(landscape reference) - Important objects →
entity_type: "visual_asset"(detail reference)
2. Describe ONLY appearance in the entity description — this generates the reference image:
"Chubby orange tabby cat with blue apron, straw hat"(what it looks like)
3. Write scene prompts as ACTION — reference entities by name, describe what they DO:
"Pippip stands behind Fish Stall, arranging fish..."(what happens)- NOT:
"A chubby orange tabby cat wearing a blue apron stands behind a wooden stall..."(don't repeat appearance)
4. List all entities that appear in each scene's character_names array — their reference images get passed to the AI as visual input, ensuring consistency.
Story idea
↓
Break into visual elements → characters[] array with entity_type + description
↓
Write scene prompts using entity NAMES → character_names lists which refs to use
↓
System generates ref image per entity → then composes scenes using those refs
Skills handle all the API calls, polling, and verification automatically. Use with Claude Code (/fk-command) or follow the recipe in skills/*.md for any AI agent.
/fk-create-project ← interactive: asks story, creates entities + scenes
/fk-gen-refs <project_id> ← generates all reference images, verifies UUIDs
/fk-gen-images <pid> <vid> ← generates scene images with all refs applied
/fk-gen-videos <pid> <vid> ← generates videos (2-5 min each, polls automatically)
/fk-concat <vid> ← downloads + merges into final video
/fk-status <pid> ← dashboard: what's done, what's next
Full pipeline in 5 commands. Each skill pre-checks dependencies (e.g. /fk-gen-images verifies all refs exist first).
Click to expand raw curl commands
From the story, identify every visual element that repeats across scenes:
| Element | entity_type | description (appearance only) |
|---|---|---|
| Pippip | character |
Chubby orange tabby cat, big green eyes, blue apron, straw hat |
| Fish Stall | location |
Rustic wooden stall, thatched roof, ice display |
| Open Market | location |
Southeast Asian market, colorful awnings, lanterns |
| Golden Fish | visual_asset |
Golden koi, shimmering scales, magical glow |
curl -X POST http://127.0.0.1:8100/api/projects \
-H "Content-Type: application/json" \
-d '{
"name": "Pippip the Fish Merchant",
"story": "Pippip is a chubby orange tabby cat who sells fish at a Southeast Asian open market. Scene 1: Morning setup. Scene 2: Staring at the golden fish. Scene 3: Eating the last fish at sunset.",
"characters": [
{"name": "Pippip", "entity_type": "character", "description": "Chubby orange tabby cat with big green eyes, blue apron, straw hat. Walks upright. Pixar-style 3D."},
{"name": "Fish Stall", "entity_type": "location", "description": "Small rustic wooden market stall with thatched bamboo roof, crushed ice display, hanging brass scale."},
{"name": "Open Market", "entity_type": "location", "description": "Bustling Southeast Asian open-air market with colorful awnings, hanging lanterns, stone walkway."},
{"name": "Golden Fish", "entity_type": "visual_asset", "description": "Magnificent golden koi fish with shimmering iridescent scales, elegant fins, slight magical glow."}
]
}'
# Save project_id from responseScene prompts reference entities by name (not description). character_names lists which reference images to apply.
# Create video
curl -X POST http://127.0.0.1:8100/api/videos \
-H "Content-Type: application/json" \
-d '{"project_id": "<PID>", "title": "Pippip Episode 1"}'
# Scene 1 (ROOT) — Pippip + Fish Stall + Open Market appear
curl -X POST http://127.0.0.1:8100/api/scenes \
-H "Content-Type: application/json" \
-d '{
"video_id": "<VID>", "display_order": 0,
"prompt": "Pippip stands behind Fish Stall, arranging fresh fish on ice. Sunrise, golden light in Open Market. Pixar 3D.",
"character_names": ["Pippip", "Fish Stall", "Open Market"],
"chain_type": "ROOT"
}'
# Scene 2 (CONTINUATION) — Golden Fish now appears
curl -X POST http://127.0.0.1:8100/api/scenes \
-H "Content-Type: application/json" \
-d '{
"video_id": "<VID>", "display_order": 1,
"prompt": "Pippip leans over Fish Stall, staring at Golden Fish on empty ice. Drooling. Open Market dark behind. Pixar 3D.",
"character_names": ["Pippip", "Fish Stall", "Golden Fish", "Open Market"],
"chain_type": "CONTINUATION", "parent_scene_id": "<scene-1-id>"
}'
# Scene 3 (CONTINUATION)
curl -X POST http://127.0.0.1:8100/api/scenes \
-H "Content-Type: application/json" \
-d '{
"video_id": "<VID>", "display_order": 2,
"prompt": "Pippip sits on stool at Fish Stall eating Golden Fish with chopsticks. SOLD OUT sign. Open Market sunset. Pixar 3D.",
"character_names": ["Pippip", "Fish Stall", "Golden Fish", "Open Market"],
"chain_type": "CONTINUATION", "parent_scene_id": "<scene-2-id>"
}'# Step 3: Generate reference images (one per entity, wait for each)
curl -X POST http://127.0.0.1:8100/api/requests \
-d '{"type": "GENERATE_CHARACTER_IMAGE", "character_id": "<CID>", "project_id": "<PID>"}'
# Poll: GET /api/requests/<RID> until status=COMPLETED
# Repeat for each entity. Verify all have UUID media_id.
# Step 4: Generate scene images
curl -X POST http://127.0.0.1:8100/api/requests \
-d '{"type": "GENERATE_IMAGE", "scene_id": "<SID>", "project_id": "<PID>", "video_id": "<VID>", "orientation": "VERTICAL"}'
# Worker blocks if any ref is missing media_id
# Step 5: Generate videos (2-5 min each)
curl -X POST http://127.0.0.1:8100/api/requests \
-d '{"type": "GENERATE_VIDEO", "scene_id": "<SID>", "project_id": "<PID>", "video_id": "<VID>", "orientation": "VERTICAL"}'
# Step 6: Download + concat
curl -s "http://127.0.0.1:8100/api/scenes?video_id=<VID>" # get video URLs
# Download each, normalize with ffmpeg, concatEvery visual element that should stay consistent gets a reference image — characters, locations, props. Each reference has a UUID media_id used in all scene generations via imageInputs.
| Entity Type | Aspect Ratio | Composition |
|---|---|---|
character |
Portrait | Full body head-to-toe, front-facing, centered |
location |
Landscape | Establishing shot, level horizon, atmospheric |
creature |
Portrait | Full body, natural stance, distinctive features |
visual_asset |
Portrait | Detailed view, textures, scale reference |
Scene prompts describe what happens, not character appearance. The reference images maintain visual consistency.
DO: "Pippip juggling fish at Fish Stall, crowd watching in Open Market"
DON'T: "Pippip the chubby orange tabby cat wearing a blue apron juggling..."
All media_id values are UUID format (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx). Never the base64 CAMS... mediaGenerationId.
Each scene has two separate prompts:
prompt— describes the still image (frame 0):"Luna steps out of rocket onto candy planet. Wide shot, sunrise."video_prompt— describes the 8s video motion with sub-clip timing and camera directions:
0-3s: Wide crane down, Luna steps out of rocket onto Candy Planet Surface. Luna gasps "It's beautiful!"
3-6s: Low angle tracking shot, Luna walks across candy ground, shallow DOF. Luna says "Everything is made of candy."
6-8s: Close-up Luna's face, eyes wide with wonder, golden hour backlight. Silence, ambient wind.
Characters can have a voice_description (max ~30 words) for voice consistency:
{"name": "Luna", "entity_type": "character", "description": "Small white cat...", "voice_description": "Soft curious childlike voice with wonder and slight purring"}Voice descriptions are auto-appended to video prompts before generation.
The worker auto-appends "No background music. Keep only natural sound effects and ambient sounds." to all video prompts. Sound effects from the scene (footsteps, splashing, wind) are preserved.
1. Create project POST /api/projects (with entities + story)
2. Create video POST /api/videos
3. Create scenes POST /api/scenes (chain_type: ROOT → CONTINUATION)
4. Gen ref images POST /api/requests {type: GENERATE_CHARACTER_IMAGE} per entity
→ Wait ALL complete, verify all have UUID media_id
5. Gen scene images POST /api/requests {type: GENERATE_IMAGE} per scene
→ Wait ALL complete
6. Gen videos POST /api/requests {type: GENERATE_VIDEO} per scene
→ Wait ALL complete (2-5 min each)
7. (Optional) Upscale POST /api/requests {type: UPSCALE_VIDEO} (TIER_TWO only)
8. Download + concat ffmpeg normalize + concat
Ready-to-use workflow recipes in skills/ (also available as /slash-commands in Claude Code):
| Skill | Description |
|---|---|
/fk-create-project |
Create project + entities + video + scenes interactively |
/fk-research |
Fact-check story details before scripting |
/fk-gen-refs |
Generate reference images for all entities |
/fk-gen-images |
Generate scene images with character refs |
/fk-gen-videos |
Generate videos from scene images (4K upscale via UPSCALE_VIDEO request, PAYGATE_TIER_TWO) |
/fk-concat |
Download + merge all scene videos |
/fk-pipeline |
Smart full-pipeline orchestrator — runs the whole chain end to end |
/fk-monitor |
Live monitor for a running pipeline |
| Skill | Description |
|---|---|
/fk-gen-chain-videos |
Auto start+end frame chaining for smooth transitions (i2v_fl) |
/fk-insert-scene |
Multi-angle shots, cutaways, close-ups within a chain |
/fk-creative-mix |
Analyze story + suggest all techniques (chain, insert, r2v, parallel) |
| Skill | Description |
|---|---|
/fk-review-video |
AI vision scoring of generated scene videos (quality, consistency, usability) — see AI Vision Providers below |
/fk-review-board |
Visual scene-by-scene review board for feedback before locking a cut |
/fk-change-provider |
View/switch the AI CLI, model and effort behind /fk-review-video |
| Skill | Description |
|---|---|
/fk-camera-guide |
Camera angles, movements, lighting, DOF for cinematic video prompts |
/fk-thumbnail-guide |
Hook-worthy thumbnail design rules |
| Skill | Description |
|---|---|
/fk-gen-tts-template |
Create a voice template for consistent narration |
/fk-import-voice |
Import an existing voice recording as a template |
/fk-gen-narrator |
Generate narrator text + TTS for all scenes |
/fk-gen-text-overlays |
Generate text overlays from narrator text (dates, locations, stats) |
/fk-concat-fit-narrator |
Trim scene videos to fit narrator duration, then concat |
/fk-gen-music |
Generate background music via Suno |
| Skill | Description |
|---|---|
/fk-youtube-seo |
Generate SEO-optimized title, description, tags |
/fk-brand-logo |
Apply channel icon watermark to video/thumbnails |
/fk-youtube-upload |
Upload to YouTube with rule validation + scheduling |
/fk-thumbnail |
Generate YouTube-optimized thumbnails |
| Skill | Description |
|---|---|
/fk-status |
Full project dashboard + recommended next action |
/fk-switch-project |
Switch the active project |
/fk-fix-uuids |
Repair any CAMS... media_ids to UUID format |
/fk-refresh-urls |
Refresh expired GCS signed URLs for images/videos |
/fk-upload-image |
Upload a local image to get a media_id |
/fk-add-material |
Image material system |
/fk-change-model |
View/switch video, image, and upscale model keys |
/fk-dashboard |
Live status in the Claude Code statusline |
/fk-doctor |
Diagnose any error (Flow API, extension, worker, YouTube) and prescribe a fix |
Skills are .md recipes any AI coding-assistant CLI can read and follow — this is about which agent reads the skill files, not which model does the work:
| CLI | Instructions | How skills work |
|---|---|---|
| Claude Code | CLAUDE.md (auto-loaded) |
Native /fk-* slash commands |
| Codex CLI | AGENTS.md → reads CLAUDE.md |
User says /fk-<name>, agent reads skills/fk-<name>.md |
The Gemini CLI target was dropped in v1.3.1 — the CLI is retired, and its
replacement agy reads none of what that target generated (see the changelog).
agy is still supported, as one of the three CLIs that can run video review —
that is configured in agent/providers.json, not by setup.py.
Separate from the table above — this is about which CLI backend does the vision analysis for /fk-review-video. Three providers are supported and swappable at runtime, no restart required:
| Provider | Binary | Reasoning efforts | Model catalog | Setup |
|---|---|---|---|---|
claude |
Claude Code CLI | low medium high xhigh max |
aliases (sonnet, opus, haiku, fable) or any full model name |
Default — works out of the box |
agy |
Google Antigravity CLI | low medium high |
closed — agy models is the whole list and agy rejects anything else |
Install separately, sign in once |
codex |
OpenAI Codex CLI | low medium high xhigh max (varies per model) |
codex's own on-disk cache, plus slugs newer than it | npm install -g @openai/codex, then codex login once |
Provider, model and effort are set per role — a role being a job an AI CLI
does for Flow Kit. There is one today, video_review; the config is a map so
the next one is an entry rather than a schema change. Model and effort may both
be null, meaning "whatever that CLI defaults to".
For agy, model and effort are mutually exclusive. Its slugs name their own
effort — gemini-3.8-flash-low, gemini-3.1-pro-high — so setting both is
rejected (--model gpt-oss-120b-medium conflicts with --effort=low), and a slug
with no effort in its name refuses --effort outright
(--effort is not supported for model "claude-sonnet-4-6"). Pick a model, or
pick an effort and let agy choose the model. The API answers 400 for the pair.
# View provider status + the current per-role config
# live=true additionally runs `<binary> --version` on each (a few seconds)
curl -s "http://127.0.0.1:8100/api/providers?live=true" | python3 -m json.tool
# List a provider's models (refresh=true bypasses the 5-minute cache)
curl -s "http://127.0.0.1:8100/api/providers/models?provider=agy" | python3 -m json.tool
# Point a role at a provider/model/effort — hot-reloaded, no server restart
curl -X PATCH http://127.0.0.1:8100/api/providers \
-H "Content-Type: application/json" \
-d '{"roles": {"video_review": {"provider": "claude", "model": "sonnet", "effort": "high"}}}'
# agy takes a model OR an effort, never both — its slugs name their own effort
curl -X PATCH http://127.0.0.1:8100/api/providers \
-H "Content-Type: application/json" \
-d '{"roles": {"video_review": {"provider": "agy", "model": "gemini-3.8-flash-low"}}}'
# The older whole-agent switch still works. It also clears each role's model
# (a slug means nothing to a different CLI) and drops an effort the new
# provider does not have.
curl -X PATCH http://127.0.0.1:8100/api/providers \
-H "Content-Type: application/json" -d '{"active": "agy"}'Or edit it in the dashboard under Settings, or run /fk-change-provider for
an interactive picker. Full details in skills/fk-change-provider.md.
codex needs credits on its OpenAI workspace. installed: true only means
the binary is on PATH; a workspace with no balance fails every review with
ERROR: Your workspace is out of credits.
Contact sheets lose their timestamps on an ffmpeg without libfreetype.
Homebrew's ffmpeg 8.x is one such build: drawtext is simply absent, and
naming a filter that does not exist aborts the whole chain. Flow Kit probes for
it once and falls back to untimestamped frames, telling the model the frame
interval instead so it can still answer in time ranges. ffmpeg -filters | grep drawtext shows whether yours has it.
| Technique | API Type | Use Case |
|---|---|---|
| i2v | GENERATE_VIDEO |
Image → video (standard) |
| i2v_fl | GENERATE_VIDEO + endImage |
Start+end frame → smooth scene transitions |
| r2v | GENERATE_VIDEO_REFS |
Reference images → video (intros, dream sequences) |
| Upscale | UPSCALE_VIDEO |
Video → 4K (TIER_TWO only) |
| Resource | Create | List | Get | Update | Delete |
|---|---|---|---|---|---|
| Project | POST /api/projects |
GET /api/projects |
GET /api/projects/{id} |
PATCH /api/projects/{id} |
DELETE /api/projects/{id} |
| Character | POST /api/characters |
GET /api/characters |
GET /api/characters/{id} |
PATCH /api/characters/{id} |
DELETE /api/characters/{id} |
| Video | POST /api/videos |
GET /api/videos?project_id= |
GET /api/videos/{id} |
PATCH /api/videos/{id} |
DELETE /api/videos/{id} |
| Scene | POST /api/scenes |
GET /api/scenes?video_id= |
GET /api/scenes/{id} |
PATCH /api/scenes/{id} |
DELETE /api/scenes/{id} |
| Request | POST /api/requests |
GET /api/requests |
GET /api/requests/{id} |
PATCH /api/requests/{id} |
— |
| Endpoint | Description |
|---|---|
GET /health |
Server + extension status |
GET /api/flow/status |
Extension connection details |
GET /api/flow/credits |
User credits + tier |
GET /api/requests/pending |
Pending request queue |
GET /api/projects/{id}/characters |
Entities linked to project |
| Type | Required Fields | Async? | reCAPTCHA? |
|---|---|---|---|
GENERATE_CHARACTER_IMAGE |
character_id, project_id | No | Yes |
GENERATE_IMAGE |
scene_id, project_id, video_id, orientation | No | Yes |
GENERATE_VIDEO |
scene_id, project_id, video_id, orientation | Yes | Yes |
GENERATE_VIDEO_REFS |
scene_id, project_id, video_id, orientation | Yes | Yes |
UPSCALE_VIDEO |
scene_id, project_id, video_id, orientation | Yes | Yes |
- Server handles throttling — worker enforces max 5 concurrent + 10s cooldown automatically. Use
POST /api/requests/batchto submit all at once; do NOT manually batch. - 10s cooldown between API calls (anti-spam, configurable via
API_COOLDOWN) - Reference blocking — scene image gen refuses if any referenced entity is missing
media_id - Skip completed — won't re-generate already-completed assets
- Cascade clear — regenerating image auto-resets downstream video + upscale
- Retry — failed requests retry up to 5 times
- UUID enforcement — extracts UUID from fifeUrl if response doesn't provide it directly
- Voice context — auto-appends character
voice_descriptionto video prompts - No background music — auto-appends "no background music, keep sound effects" to all video prompts
- Dual video response schema — Lite/Fast/Ultra models return
operations[]and stream URLs; Low Priority models (veo_3_1_*_low_priority,*_ultra_relaxed) returnworkflows + mediawith the MP4 inline as base64. The SDK auto-detects, validates theftypmagic, and saves the binary tooutput/_workflow_videos/{media_id}.mp4. The scene's_video_urlis then afile://path whichcurlandffmpeghandle natively._video_media_idalways stores the real Flow media UUID, so upscale works for both schemas.
The default for PAYGATE_TIER_TWO frame_2_video and start_end_frame_2_video is veo_3_1_i2v_lite_low_priority — the TRUE 0-credit Low Priority that works on every service tier including SERVICE_TIER_ADVANCED.
The *_ultra_relaxed family (Low Priority ultra-quality) silently returns empty operations on SERVICE_TIER_ADVANCED accounts because Google requires SERVICE_TIER_ULTRA for that path. ULTRA-tier users can switch back via /fk-change-model — see skills/fk-change-model.md for the full preset list and tier compatibility matrix.
Every project must have a material field that controls the visual style of generated images. Set it at project creation.
# List available materials
curl -s http://127.0.0.1:8100/api/materials
# Set on project
curl -X POST http://127.0.0.1:8100/api/projects \
-d '{"name": "...", "material": "3d_pixar", ...}'Materials control both entity image_prompt style and scene scene_prefix. Examples: realistic, 3d_pixar, anime, stop_motion, minecraft, oil_painting.
| Variable | Default | Description |
|---|---|---|
API_HOST |
127.0.0.1 |
REST API bind address |
API_PORT |
8100 |
REST API port |
WS_HOST |
127.0.0.1 |
WebSocket server bind |
WS_PORT |
9222 |
WebSocket server port |
POLL_INTERVAL |
5 |
Worker poll interval (seconds) |
MAX_RETRIES |
5 |
Max retries per request |
VIDEO_POLL_TIMEOUT |
420 |
Video gen poll timeout (seconds) |
API_COOLDOWN |
10 |
Seconds between API calls (anti-spam) |
agent/
├── main.py # FastAPI app + WebSocket server
├── config.py # Configuration (loads models.json, providers.json)
├── models.json # Video/upscale/image model mappings
├── providers.json # Per-role AI CLI provider/model/effort (claude/agy/codex)
├── db/
│ ├── schema.py # SQLite schema (aiosqlite)
│ └── crud.py # Async CRUD with column whitelisting
├── models/ # Pydantic models + Literal enums
├── api/ # REST routes (projects, videos, scenes, characters, requests,
│ # flow, models, providers, reviews, materials, music, tts)
├── services/
│ ├── flow_client.py # WS bridge to extension
│ ├── tts.py # OmniVoice TTS (subprocess-based)
│ ├── scene_chain.py # Continuation scene logic
│ ├── cli_providers.py # What each AI CLI accepts; role → provider/model/effort
│ ├── video_reviewer.py # AI vision review — contact sheet + claude/agy/codex CLI dispatch
│ └── post_process.py # ffmpeg trim/merge/music
└── worker/
└── processor.py # Queue processor + poller
extension/ # Chrome MV3 extension
skills/ # AI agent workflow recipes (CLI-agnostic)
youtube/
├── auth.py # OAuth2 multi-channel auth
├── upload.py # Upload with scheduling + rule validation
└── channels/ # Per-channel config (gitignored)
└── <channel_name>/
├── client_secrets.json # OAuth2 credentials
├── token.json # Auth token (auto-created)
├── channel_rules.json # Upload rules + SEO defaults
└── upload_history.json # Upload log
CLAUDE.md # AI agent instructions (Claude Code)
AGENTS.md # AI agent instructions (Codex CLI, generated)
Optional narrator voice for scenes. Uses OmniVoice — multilingual zero-shot TTS with voice cloning (600+ languages).
See skills/fk-gen-tts-template.md for full install guide. Quick version:
pip install torch==2.8.0 torchaudio==2.8.0 # or +cu128 for NVIDIA
pip install omnivoice
python3 -c "from omnivoice import OmniVoice; print('OK')"If OmniVoice is in a separate venv, point to it:
export TTS_PYTHON_BIN=/path/to/omnivoice-venv/bin/python3- Create voice template —
/fk-gen-tts-template— generates an anchor voice WAV - Add narrator text to scenes —
PATCH /api/scenes/{id}withnarrator_text - Generate narration —
/fk-gen-narrator— voice-clones the template for each scene - Concat with narration —
/fk-concat-fit-narrator— trims scene videos to match TTS duration
CPU-only recommended (MPS produces artifacts). ~15-30s per scene.
Automated upload with per-channel rules, SEO optimization, and brand watermarking.
# 1. Place OAuth credentials
cp client_secrets.json youtube/channels/<channel_name>/
# 2. Authenticate (opens browser)
python3 youtube/auth.py <channel_name> # Linux / Windows (WSL)
arch -arm64 python3 youtube/auth.py <channel_name> # macOS Apple Silicon
# 3. Token saved to youtube/channels/<channel_name>/token.json (auto-refreshes)Each channel has a rules file controlling upload scheduling and SEO:
{
"shorts": {"max_per_day": 3, "optimal_times": ["07:00", "12:00", "17:00"]},
"long_form": {"max_per_day": 1, "optimal_times": ["19:00"]},
"scheduling": {"min_gap_hours": 4, "avoid_hours": [0,1,2,3,4,5]},
"seo": {"niche": "...", "default_tags": [...], "title_max_chars": 65}
}/fk-youtube-seo → generates title, description, hashtags, tags
/fk-brand-logo → applies channel icon watermark
/fk-youtube-upload → validates rules + uploads (auto-detects Short vs Long-form)
Upload validation checks: max per day, min gap between uploads, avoid dead hours. Auto-detects Short (<61s + vertical 9:16) vs Long-form.
Errors can originate from four layers — Google Flow backend, Chrome extension, FastAPI layer, and the worker itself. The worker's _handle_failure (agent/worker/processor.py:414-481) routes recovery by error_message string content, not HTTP status, because Flow lumps many distinct failures under HTTP 400 with varying details.reason values.
These arrive in the response body as data.error.details[].reason. The worker appends the reason to error_message as "<msg> [<reason>]".
| Reason string | Meaning | Auto-handling |
|---|---|---|
PUBLIC_ERROR_UNSAFE_GENERATION |
Prompt tripped safety filter (people, violence, nudity) | Mark FAILED — rewrite prompt (use alias names, remove triggers) |
PUBLIC_ERROR_USER_QUOTA_REACHED |
Daily credits exhausted | Mark FAILED — wait for reset or upgrade tier |
PUBLIC_ERROR_MODEL_ACCESS_DENIED |
Tier mismatch (e.g. TIER_ONE trying Veo 3 / upscale) | Mark FAILED — auto-detect should downgrade to allowed model |
Requested entity was not found |
Uploaded media_id expired (~1h TTL) |
Auto-recover via _recover_entity_not_found — re-uploads from image_url, re-queues PENDING |
Internal error encountered |
Flow backend transient 500 | Exponential backoff retry: 2^retry * 10s, capped 300s |
reCAPTCHA failed / captcha |
Extension couldn't solve CAPTCHA | Retry up to 10× without incrementing retry_count (processor.py:454-464) |
PUBLIC_ERROR_UNUSUAL_ACTIVITY (403, message reCAPTCHA evaluation failed) |
Google flagged the session as bot-like — usually rapid bursts of submits, VPN/shared IP, or stale auth cookies | NOT auto-recoverable. Pause submits, clear cookies for google.com + labs.google in Chrome, sign back in at flow.google.com, then resubmit with ≥1s gap and ≤5 concurrent. See /fk-doctor for full playbook. |
| Status | Source | Meaning | Handling |
|---|---|---|---|
| 400 | Flow API | Invalid payload, UNSAFE_GENERATION, entity not found (sometimes) | Route by details.reason — some are auto-recoverable, others terminal |
| 401 | Flow API | Should not occur — batchexecute authenticates in the page, not with a bearer | Check the Flow tab is signed in; see NO_AT_TOKEN |
| 403 | Extension (background.js:432) |
CAPTCHA_FAILED, NO_FLOW_TAB, or MODEL_ACCESS_DENIED |
CAPTCHA → retry loop; NO_FLOW_TAB → fail (user must open Flow); tier → fail |
| 404 | Flow API | media_id not found (expired upload) |
Same as "Requested entity was not found" — auto re-upload |
| 429 | Flow API | Rate limited / quota | Back off + retry; if USER_QUOTA_REACHED appears, fail |
| 500 | Flow backend or extension fetch exception (background.js:504) |
Transient server error OR network drop during fetch | Retry with exponential backoff |
| 502 | FastAPI default (agent/api/flow.py:80,92) |
Extension returned error without explicit status | Retry; check extension health |
| 503 | FastAPI (api/flow.py) |
"Extension not connected" | Worker waits for reconnect — status set to PENDING, not FAILED |
| 504 | Agent | 60s timeout waiting for extension WS response | Treated as transient; re-queue PENDING |
Status-code detection logic lives in agent/worker/_parsing.py:_is_error — a result is an error if result.error is set, status >= 400, or data.error is present.
String patterns in error_message that the worker recognizes:
| Error message contains | Cause | Handling |
|---|---|---|
Extension not connected |
Chrome extension offline or WS dropped | 503 returned; worker re-queues PENDING and waits |
extension reconnected / extension disconnected |
WS bounce mid-request | Re-queue PENDING without incrementing retry_count |
extension_switched |
User switched Flow tabs mid-generation | Re-queue PENDING |
NO_AT_TOKEN |
Flow tab is signed out, on an interstitial, or still booting | Open flow.google.com, sign in, let the app load |
NO_FLOW_PROJECT |
No Flow project to scope the RPC to | Pin FLOW_PROJECT_ID — terminal, not retried |
UNSUPPORTED_ON_BATCH_API |
Upscale / r2v / chaining — payload never captured | See docs/CAPTURE.md — terminal, not retried |
NO_FLOW_TAB |
No Google Flow tab available for reCAPTCHA | User must open a Flow tab |
Failed to fetch |
Network drop inside extension service worker | Retry with backoff |
timeout / WS 60s no response |
Extension hung mid-request | Re-queue PENDING |
processor.py:_handle_failure decides terminal vs retryable:
- Auto-recover if message contains
"not found"→ re-upload media, mark PENDING. - Transient WS (
reconnected/disconnected/switched) → re-queue PENDING, keepretry_count. - CAPTCHA → retry up to 10× without counting toward
MAX_RETRIES. - Default → increment
retry_count; if <MAX_RETRIES(5), schedule retry with2^retry * 10sbackoff (capped 300s). Otherwise mark FAILED.
From youtube/upload.py (HTTP errors from YouTube Data API v3):
| Error | Cause | Fix |
|---|---|---|
invalidTags (400) |
Tags exceed 500-char limit (incl. quote overhead: spaces → +2 per tag) | Trim tags; validate with sum(len(t) + (2 if ' ' in t else 0) for t in tags) + (len(tags)-1) <= 500 |
invalidCategoryId (400) |
Unknown category | Use "22" (People & Blogs) or "24" (Entertainment) |
quotaExceeded (403) |
Daily 10K quota exhausted (uploads cost 1600) | Wait 24h (Pacific midnight reset) |
uploadLimitExceeded (400) |
Channel daily upload cap hit | Wait 24h or use different channel |
invalid_grant (auth) |
Token revoked or expired | Re-run python3 youtube/auth.py <channel> |
scheduledPublishTimeInPast |
publishAt <= now |
Use auto_schedule() or bump to next day |
| Problem | Solution |
|---|---|
| Extension shows "Agent disconnected" | Start python -m agent.main |
| Extension shows "No token" | Expected on the batch path — there is no bearer token any more |
CAPTCHA_FAILED: NO_FLOW_TAB |
Open a Google Flow tab |
403 MODEL_ACCESS_DENIED |
Tier mismatch — check /api/flow/credits, downgrade model in models.json |
403 PUBLIC_ERROR_UNUSUAL_ACTIVITY / reCAPTCHA evaluation failed |
Pause submits, clear cookies for google.com + labs.google in Chrome, sign back in, then resubmit with ≥1s gap and ≤5 concurrent. Switch network or wait 1–6 h if still blocked |
| Scene images inconsistent | Check all refs have UUID media_id — run /fk-fix-uuids |
media_id starts with CAMS... |
Run /fk-fix-uuids to extract UUID from URL |
| Upscale "permission denied" | Requires PAYGATE_TIER_TWO account |
| Request stuck in PROCESSING | Check error_message history; if extension dropped, restart extension |
| "Requested entity was not found" spam | Image URLs expired — re-upload via POST /api/upload-image or wait for auto-recovery |
YouTube upload invalidTags |
Tag-char overflow; reduce tags (quote overhead bytes count) |
Python cryptography arch mismatch |
Use python3.10, not python3.13 (x86/arm64 binary mismatch) |
Dates are merge dates. Older releases are tagged; git log is the full record.
| Date | Change |
|---|---|
| 2026-09-20 | setup.py --tool gemini removed. It generated .gemini/commands/fk/*.toml and GEMINI.md for a CLI that is retired, and its replacement agy reads neither — verified against agy 1.2.7: a project's .gemini/commands/*.toml is not expanded, .claude/commands/*.md is not either, and GEMINI.md, AGENTS.md and CLAUDE.md are all absent from a print-mode run's context even inside a trusted folder. agy plugin import imports extensions, not command files. GEMINI.md is deleted; setup.py clean still removes what the target left on disk, because nothing else ever will. agy remains fully supported — as one of the three CLIs that run video review, configured in agent/providers.json rather than by setup.py |
| 2026-09-20 | AGENTS.md is genuinely generated again. It says "do not edit" and had been edited anyway: rules 14-16 (fact-check, real-people bypass, review-before-upscale) and pipeline steps 0 and 7.5 lived only in the committed artifact, so setup.py sync would have deleted three operational rules. They are in setup.py now. The same drift had left the skill table listing 25 skills — missing 11 that exist and naming one that does not; it is rebuilt from skills/ at generation time |
| 2026-09-20 | setup.py gains its first tests (13), including that a .fk-setup.json written before this release — which records "gemini" — is skipped with a pointer to clean instead of crashing the sync |
Video review had been failing on every path at once, which is why nothing about
it looked fixable from the symptoms. The unit suite was red on a normal dev
machine for the whole period — 13 of these tests fail on v1.2.0 — while CI
stayed green, because the workflow installs ffmpeg and a font and asserts
drawtext renders. The one environment that ran the suite was the one
environment where it worked.
All three providers are verified end-to-end against the real CLIs: a synthetic clip with a planted mid-clip defect, through frame extraction, contact sheets and a live vision call, with no mocks. claude, agy and codex each find the defect and score it, on their default model and on an explicitly selected model + effort.
| Date | Change |
|---|---|
| 2026-09-20 | ffmpeg without drawtext: Homebrew's ffmpeg 8.x is built without libfreetype, so the timestamp filter does not exist and naming it aborted the whole chain — frame extraction died before any provider was reached. Probed once, with a fallback to untimestamped frames and a prompt that hands the model the frame interval instead |
| 2026-09-20 | agy was auto-denied: handed a bare file path, agy reaches for a shell command to look at the file, headless mode cannot prompt for that permission, and the run returns an empty response on a zero exit code. Fixed by steering it at its own file-reading tool and naming the sheet directory with --add-dir, which gets the read done unprivileged — so --dangerously-skip-permissions, which auto-approves every tool including arbitrary shell commands, is gone. Output is parsed from --output-format json, and a denied tool is an error whether or not agy still answered — the prompt carries the rubric and both scene prompts, so a denied run can write a plausible review from the text alone |
| 2026-09-20 | codex no longer bypasses its sandbox: -i hands codex the image bytes directly, so the run needs neither a shell nor a writable filesystem. --dangerously-bypass-approvals-and-sandbox bought nothing and cost the sandbox; --sandbox read-only already implies approval: never. An empty output file is now an error instead of a JSON decode failure three frames away |
| 2026-09-20 | A malformed error entry no longer vanishes. The parser required the exact keys severity/time_range/description and silently dropped anything else — and what it dropped was usually CRITICAL, the one severity that caps character_consistency at 3.0 and forces the verdict below acceptable, so timeRange instead of time_range turned an unusable video into a clean pass. The three fields are now handled by what they can cost: near-miss names are normalised, a missing time range or description is repaired and logged, and only a severity outside {CRITICAL, HIGH, MINOR} fails the scene — that is the one field with no safe default, because without it we do not know whether the video passed. VideoError.severity is a Literal now, so the three code paths that branch on it cannot be handed anything else |
| 2026-09-20 | A review with no scores in it is now a failure, not a score. Every dimension defaults to 5.0, so a CLI answer carrying no dimensions became a complete, plausible review — 5.0 across the board, verdict "poor", zero errors — of a video nothing had actually looked at |
| 2026-09-20 | stdin closed for all three CLIs. Each appends piped stdin to the prompt when stdin is not a terminal — codex documents it as a <stdin> block. Under uvicorn that is whatever the launching shell handed down |
| 2026-09-20 | Per-role provider, model and effort, editable in the dashboard under Settings or via PATCH /api/providers. Efforts are validated against each CLI's real ladder (agy stops at high); models are validated only for agy, whose catalog is closed, so a slug newer than claude's or codex's cache still goes through. A whole-agent {"active": …} switch clears each role's model and clamps its effort, because neither survives a change of CLI |
| 2026-09-20 | Review hardening from an adversarial pass: one review is pinned to one provider (the role was resolved per scene, so a hand edit or a dashboard poll mid-run could split a video's score across two backends); an unknown agy model is a 400 naming the known slugs instead of a failed review; CLI failures carry stdout as well as stderr (claude puts its readable sentence there); providers.json is written atomically (a truncated file hard-fails config.py at import, so the server would not boot); and the drawtext probe is an optimisation now — extraction retries untimestamped if the filter is listed but cannot render, which is what an ffmpeg with libfreetype and no font does |
| 2026-09-20 | agy's model and effort are mutually exclusive and the config now says so. Its slugs name their own effort (gemini-3.8-flash-low), so the pair is rejected — a mismatch conflicts, and a slug with no effort in its name refuses --effort at all |
Flow moved to flow.google.com in September 2026 and stopped minting the bearer
token the old REST API needed. Everything below is that migration.
| Date | Change |
|---|---|
| 2026-09-18 | /health reports the app's real version again — it had been pinned at 0.2.0 since v0.2.0 while the app said 1.1.0, because only one of the two literals was ever bumped (#52) |
| 2026-09-18 | Omni 1.1 Flash first-frame, first+last and reference modes ported to batchexecute (#48, #50). Unported capabilities drop from four to three, all on the Veo path |
| 2026-09-17 | REST transport removed — the ten _legacy_* methods, the USE_BATCH_RPC branches, the fingerprint pools and agent/services/headers.py; net −1043 lines (#49) |
| 2026-09-17 | Migrated image API: variant submit, settled-wave retry, and 2K/4K image export via SPrCad (#42) |
| 2026-09-15 | Video submit unified across the frame and reference paths (#46) |
| 2026-09-15 | Omni Flash text-to-video on the batch path (#41) |
| 2026-09-15 | Extension: idle-tab leak fixed (#44) |
| 2026-09-07 | batchexecute transport added — the agent builds the envelope, the extension signs it inside a signed-in Flow tab (#39) |
| Date | Change |
|---|---|
| 2026-08-18 | Omni Flash generation (#30); MV3 flow-key bootstrap (#24) |
| 2026-08-04 | Web dashboard rebuilt with the real pipeline UI, a guide page and i18n |
| 2026-08-04 | Video review: contact sheets split by duration rather than one giant tile; agy and codex added as review CLI providers |
| 2026-08-04 | Skill files standardised on fk-<name> |
| 2026-05-09 | v1.1.0 |
| 2026-04-27 | v1.0.2 |
| 2026-04-22 | v1.0.1 |
MIT
Share anything crazy and useful created with Vibe Code. Drop in to:
- Post the story-video runs and thumbnails you've generated
- Share scene templates, prompt recipes, and reference-image setups
- Ask for help when an output isn't matching what you imagined
- Request features and report bugs you've hit in the wild
- Trade tips on Google Flow plan limits, Veo i2v behaviour, and Chrome extension setup
- Facebook Post via Extension MCP
- Right way to build Mobile Application + System
