Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024)
| Date | Stars |
|---|---|
| 2026-07-31 | 679 |
| 2026-08-03 | 679 |
| 2026-08-06 | 679 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
## <img src="demo/assistant_rectangle.png" height="25"> VideoLLM-online: Online Video Large Language Model for Streaming Video <a href="https://showlab.github.io/videollm-online/" target="_blank"><img alt="Homepage" src="https://img.shields.io/badge/🌍 Homepage-d35400?color=d35400" /></a> <a href="https://huggingface.co/spaces/chenjoya/videollm-online" target="_blank"><img alt="Demo" src="https://img.shields.io/badge/🤗 Hugging Face Spaces-ffc107?color=ffc107" /></a> <a href="https://arxiv.org/abs/2406.11816" target="_blank"><img alt="Paper" src="https://img.shields.io/badge/📄 Paper-28a745?color=28a745" /></a> <a href="https://huggingface.co/chenjoya/videollm-online-8b-v1plus" target="_blank"><img alt="Checkpoint" src="https://img.shields.io/badge/🤗 Hugging Face Models-2980b9?color=2980b9" /></a> <a href="https://huggingface.co/datasets/chenjoya/videollm-online-chat-ego4d-134k" target="_blank"><img alt="Data" src="https://img.shields.io/badge/🤗 Hugging Face Datasets-8e44ad?color=8e44ad" /></a> ## Updates ### 2025-09-03 I made some updates to make sure this repo can be inference with the latest transformers==4.55.4 and pytorch==2.7.1+cu128. Similar versions should be okay. If you want to obtain a strong VLM-online model, I highly recommend you to finetune Qwen2.5VL-Instruct with the streaming EOS loss here. No need to use the weak text-only Llama3 again. ### 2025-04-24 Updates! Our new paper: [LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale](https://huggingface.co/papers/2504.16030). All resources, including the training video data, have been released at [LiveCC Webpage](https://showlab.github.io/livecc/) ### TLDR The first streaming video LLM, high speed (5 ~ 10 FPS on NVIDIA 3090 GPU, 10 ~ 15 FPS on A100GPU) on long-form videos (10 minutes), with SOTA performance on online/offline settings. [](https://www.youtube.com/watch?v=09OoQwxQBoc) ### Introduction This is the official implementation of [VideoLLM-online: Online Video Large Language Model for Streaming Video](https://arxiv.org/abs/2406.11816), CVPR 2024. Our paper introduces several interesting stuffs compared to popular image/video/multimodal models: - **Online Video Streaming**: Unlike previous models that serve as offline mode (querying/responding to a full video), our model supports **online interaction within a video stream**. It can **proactively** update responses during a stream, such as recording activity changes or helping with the next steps in real time. Even GPT-4o, which is audio-driven, requires user voice interaction with the visual scene, not actual video streaming. - **Cheap and Scalable Streaming Data Synthesis**: Current video datasets for training multimodal LLMs are mostly offline and unsuitable for training an online video language model. Our method **transforms any offline annotation into streaming dialogue data** by prompting open-source LLM. The model is entirely trained on Llama synthesized data. - **Parallelized Real-Time Inference**: Our inference method **parallelizes** video encoding, LLM forwarding for video frames, and LLM response generation, arranging them asynchronously. This significantly enhances real-time performance, achieving 10-15 FPS on an A100 GPU. ### Quick Start - (Recommended) Launch the gradio demo locally with: ```sh python -m demo.app --resume_from_checkpoint chenjoya/videollm-online-8b-v1plus ``` But if there are some bugs with flash-attn, try to use ``` python -m demo.app --resume_from_checkpoint chenjoya/videollm-online-8b-v1plus --attn_implementation sdpa ``` - (Recommended) Launch the CLI locally with: ```sh python -m demo.cli --resume_from_checkpoint chenjoya/videollm-online-8b-v1plus ``` - (Deprecated, HF Spaces too slow) Try demo at <a href="https://huggingface.co/spaces/chenjoya/videollm-online" target="_blank"> <img alt="Demo" src="https://img.shields.io/badge/🤗 Hugging Face Spaces-ffc107?color=ffc107" /> </a>
Excerpt of 7,482 characters
Read on GitHub
Joya Chen · National University of Singapore
58
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:8e841361eca187fb, llm:Repository title and description: 'VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024)'. Focus on video LLM for streaming video (computer vision + multimodal LLM).
matched fp:8e841361eca187fb, llm:Repository title and description: 'VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024)'. Focus on video LLM for streaming video (computer vision + multimodal LLM).
matched fp:8e841361eca187fb, llm:Repository title and description: 'VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024)'. Focus on video LLM for streaming video (computer vision + multimodal LLM).
matched fp:8e841361eca187fb, llm:Repository title and description: 'VideoLLM-online: Online Video Large Language Model for Streaming Video (CVPR 2024)'. Focus on video LLM for streaming video (computer vision + multimodal LLM).