Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Analyze videos using LLMs, Computer Vision and Automatic Speech Recognition
| Date | Stars |
|---|---|
| 2026-07-31 | 1526 |
| 2026-08-06 | 1533 |
Today
+7 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Video Analysis using vision models like Llama3.2 Vision and OpenAI's Whisper Models A video analysis tool that combines vision models like Llama's 11B vision model and Whisper to create a description by taking key frames, feeding them to the vision model to get details. It uses the details from each frame and the transcript, if available, to describe what's happening in the video. ## Table of Contents - [Features](#features) - [Requirements](#requirements) - [System Requirements](#system-requirements) - [Installation](#installation) - [Ollama Setup](#ollama-setup) - [OpenAI-compatible API Setup](#openai-compatible-api-setup-optional) - [Usage](#usage) - [Quick Start](#quick-start) - [Sample Output](#sample-output) - [Complete Usage Guide](docs/USAGES.md) - [Design](#design) - [Detailed Design Documentation](docs/DESIGN.md) - [Project Structure](#project-structure) - [Configuration](#configuration) - [Output](#output) - [Prompt Tuning](#prompt-tuning) - [Uninstallation](#uninstallation) - [License](#license) - [Contributing](#contributing) ## Features - 💻 Can run completely locally - no cloud services or API keys needed - ☁️ Or, leverage any OpenAI API compatible LLM service (openrouter, openai, etc) for speed and scale - 🎬 Intelligent key frame extraction from videos - 🔊 High-quality audio transcription using OpenAI's Whisper - 👁️ Frame analysis using Ollama and Llama3.2 11B Vision Model - 📝 Natural language descriptions of video content - 🔄 Automatic handling of poor quality audio - 📊 Detailed JSON output of analysis results - ⚙️ Highly configurable through command line arguments or config file ## Design The system operates in three stages: 1. Frame Extraction & Audio Processing - Uses OpenCV to extract key frames - Processes audio using Whisper for transcription - Handles poor quality audio with confidence checks 2. Frame Analysis - Analyzes each frame using vision LLM - Each analysis includes context from previous frames - Maintains chronological progression - Uses frame_analysis.txt prompt template 3. Video Reconstruction - Combines frame analyses chronologically - Integrates audio transcript - Uses first frame to set the scene - Creates comprehensive video description  ## Requirements ### System Requirements - Python 3.11 or higher - FFmpeg (required for audio processing) - When running LLMs locally (not necessary when using openrouter) - At least 16GB RAM (32GB recommended) - GPU at least 12GB of VRAM or Apple M Series with at least 32GB ### Installation 1. Clone the repository: ```bash git clone https://github.com/byjlw/video-analyzer.git cd video-analyzer ``` 2. Create and activate a virtual environment: ```bash python3 -m venv .venv source .venv/bin/activate # On Windows: .venv\Scripts\activate ``` 3. Install the package: ```bash pip install . # For regular installation # OR pip install -e . # For development installation ``` 4. Install FFmpeg: - Ubuntu/Debian: ```bash sudo apt-get update && sudo apt-get install -y ffmpeg ``` - macOS: ```bash brew install ffmpeg ``` - Windows: ```bash choco install ffmpeg ``` ### Ollama Setup 1. Install Ollama following the instructions at [ollama.ai](https://ollama.ai) 2. Pull the default vision model: ```bash ollama pull llama3.2-vision ``` 3. Start the Ollama service: ```bash ollama serve ``` ### OpenAI-compatible API Setup (Optional) If you want to use OpenAI-compatible APIs (like OpenRouter or OpenAI) instead of Ollama: 1. Get an API key from your provider: - [OpenRouter](https://openrouter.ai) - [OpenAI](https://platform.openai.com) 2. Configure via command line: ```bash # For OpenRouter video-analyzer video.mp4 --client openai_api --api-key your-key --api-url https://openrouter.ai/api/v1 --model gpt-4o # For OpenAI video-analyzer video.mp4 --client openai_api --api-key your-key --api-url https://api.openai.com/v1 --model gpt
Excerpt of 6,953 characters
Read on GitHub
44
Kyle Howells
1
1
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:c5008f584ec9f6ed, topic:asr, desc:speech recognition