Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (ECCV 2024)
| Date | Stars |
|---|---|
| 2026-07-31 | 861 |
| 2026-08-06 | 861 |
| 2026-08-30 | 862 |
| 2026-09-04 | 861 |
| 2026-09-20 | 861 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models <a href='https://llama-vid.github.io/'><img src='https://img.shields.io/badge/Project-Page-Green'></a> <a href='http://103.170.5.190:7864/'><img src='https://img.shields.io/badge/Project-Demo-violet'></a> <a href='https://arxiv.org/abs/2311.17043'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a> <a href='https://huggingface.co/YanweiLi'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue'></a> <a href='https://huggingface.co/datasets/YanweiLi/LLaMA-VID-Data'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Data-green'></a> LLaMA-VID empowers existing frameworks to support hour-long videos and pushes their upper limit with an extra context token. We build this repo based on LLaVA. ## Release - [24/07/04] 🔥 Our work has been accepted to ECCV 2024! - [23/12/05] 🔥 We release the full training and evalution [model](https://huggingface.co/YanweiLi/llama-vid-7b-full-224-long-video), [data](https://huggingface.co/datasets/YanweiLi/LLaMA-VID-Data), and scripts to support movie chating! - [23/11/29] 🔥 LLaMA-VID is comming! We release the [paper](https://arxiv.org/abs/2311.17043), [code](https://github.com/dvlab-research/LLaMA-VID), [data](https://huggingface.co/datasets/YanweiLi/LLaMA-VID-Data), [models](https://huggingface.co/YanweiLi), and [demo](https://llama-vid.github.io/) for LLaMA-VID! ## Contents - [Demo](#demo) - [Install](#install) - [Model](#model) - [Preparation](#preparation) - [Train](#train) - [Evaluation](#evaluation) - [Examples](#examples) - [Citation](#citation) - [Acknowledgement](#acknowledgement) - [License](#license) ## Demo We provide some selected examples in this section. More examples can be found in our [project page](https://llama-vid.github.io/). Feel free to try our online [demo](https://llama-vid.github.io/)! <div align=center> <img width="100%" src="demos/demo.png"/> </div> ## Install Please follow the instructions below to install the required packages. 1. Clone this repository ```bash git clone https://github.com/dvlab-research/LLaMA-VID.git ``` 2. Install Package ```bash conda create -n llamavid python=3.10 -y conda activate llamavid cd LLaMA-VID pip install --upgrade pip # enable PEP 660 support pip install -e . ``` 3. Install additional packages for training cases ```bash pip install ninja pip install flash-attn --no-build-isolation ``` ## Model LLaMA-VID simply contains three parts: encoder and decoder are adopted to produce visual embedding and text-guided features, respectively; context token and content token are transformed with the tailored token generation strategy; instruction tuning is designed to unleash the potential of LLMs for image and video. <div align=center> <img width="100%" src="images/overview.png"/> </div> We provide all our fully finetuned models on Stage 1 and 2 data (Long Video + Stage 3) for LLaMA-VID: | Type | Image Size | Max Token | Base LLM | Vision Encoder | Finetuning Data | Finetuning schedule | Download | |----------|----------|----------|----------|----------------|---------------|--------------------|------------------| Image only | 224 | 4K | Vicuna-7B-v1.5 | EVA-G | LLaVA1.5-Instruct | full_ft-1e | [ckpt](https://huggingface.co/YanweiLi/llama-vid-7b-full-224) | Image only | 336 | 4K | Vicuna-7B-v1.5 | EVA-G | LLaVA1.5-Instruct | full_ft-1e | [ckpt](https://huggingface.co/YanweiLi/llama-vid-7b-full-336) | Image only | 336 | 4K | Vicuna-13B-v1.5 | EVA-G | LLaVA1.5-Instruct | full_ft-1e | [ckpt](https://huggingface.co/YanweiLi/llama-vid-13b-full-336) | Short video | 224 | 4K | Vicuna-7B-v1.5 | EVA-G | LLaVA1.5-VideoChatGPT-Instruct | full_ft-1e | [ckpt](https://huggingface.co/YanweiLi/llama-vid-7b-full-224-video-fps-1) | Short video | 224 | 4K | Vicuna-13B-v1.5 | EVA-G | LLaVA1.5-VideoChatGPT-Instruct | full_ft-1e | [ckpt](https://huggingface.co/YanweiLi/llama-vid-13b-full-224-video-fps-1) | Long video | 224 | 64K | Vicu
Excerpt of 21,001 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:80916ea5cb4832cb, llm:Repository title/description: "LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (ECCV 2024)" — indicates multimodal work integrating images with LLaMA. Language: Python.
matched fp:80916ea5cb4832cb, llm:Repository title/description: "LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (ECCV 2024)" — indicates multimodal work integrating images with LLaMA. Language: Python.
matched fp:80916ea5cb4832cb, llm:Repository title/description: "LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (ECCV 2024)" — indicates multimodal work integrating images with LLaMA. Language: Python.
