Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[CVPR 2024] TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
| Date | Stars |
|---|---|
| 2026-07-31 | 425 |
| 2026-08-03 | 425 |
| 2026-08-06 | 425 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<p align="center" width="100%">
<a target="_blank"><img src="figs/logo.png" alt="TimeChat" style="width: 40%; min-width: 150px; display: block; margin: auto;"></a>
</p>
<h2 align="center"> <a href="https://arxiv.org/abs/2312.02051">TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding</a></h2>
<h4 align="center"> <a href="https://renshuhuai-andy.github.io/">Shuhuai Ren*</a>, <a href="https://yaolinli.github.io/">Linli Yao*</a>, <a href="https://lscpku.github.io/">Shicheng Li</a>, <a href="https://xusun26.github.io/">Xu Sun</a>, <a href="https://houlu369.github.io/">Lu Hou</a></h4>
<div style='display:flex; gap: 0.25rem; '>
<a href='https://arxiv.org/abs/2312.02051'><img src='https://img.shields.io/badge/Paper-PDF-red'></a>
<a href='https://huggingface.co/datasets/ShuhuaiRen/TimeIT'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-blue'></a>
<a href='https://huggingface.co/ShuhuaiRen/TimeChat-7b'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Checkpoint-blue'></a>
</div>
## News
- [25.05.01]  We release [TimeChat-Online-7B](https://timechat-online.github.io/), a new VideoLLM for efficient streaming video understanding! Now with [paper](https://arxiv.org/abs/2504.17343), [code](https://github.com/yaolinli/TimeChat-Online), [dataset](https://huggingface.co/datasets/yaolily/TimeChat-Online-139K), and [model](https://huggingface.co/wyccccc/TimeChatOnline-7B) all available! 🎉
- [24.06.04]  Add FAQ, see [FAQ.md](./docs/FAQ.md).
- [24.06.04]  Release zero-shot evaluation results of TimeChat-7b on several VideoLLM benchmarks (e.g., VideoMME, MVBench, TempCompass, etc.), see [EVAL.md](./docs/EVAL.md).
- [24.01.09] Release **TimeChat-7b** 🤗 [checkpoint](https://huggingface.co/ShuhuaiRen/TimeChat-7b) and [local demo](./demo.ipynb).
- [23.12.27] 🤗 Release the instruction-tuning dataset of **[TimeIT](https://huggingface.co/datasets/ShuhuaiRen/TimeIT)**.
- [23.12.06] Release the initial version of **TimeChat**.
<p align="center" width="100%">
<a target="_blank"><img src="figs/arch.png" alt="Video-LLaMA" style="width: 80%; min-width: 200px; display: block; margin: auto;"></a>
</p>
## Introduction
- **TimeChat** is a time-sensitive multimodal large language model specifically designed for long video understanding. Our model incorporates two key architectural contributions:
- (1) a timestamp-aware frame encoder that binds visual content with the timestamp of each frame
- (2) a sliding video Q-Former that produces a video token sequence of varying lengths to accommodate videos of various durations.
- We also construct an instruction-tuning dataset named **TimeIT**, encompassing 6 tasks and a total of 125K instances, to further enhance TimeChat's instruction-following performance.
## Example Outputs
- **An illustration of temporal localization capability of TimeChat**
<p float="left">
<img src="figs/teaser.png" style="width: 100%; margin: auto;">
</p>
- **Examples for dense video captioning (left), temporal video grounding (middle), and video highlight detection (right)**
<p float="left">
<img src="figs/case_dvc.png" style="width: 32%; margin: auto;">
<img src="figs/case_tvg.png" style="width: 34%; margin: auto;">
<img src="figs/case_vhd.png" style="width: 32%; margin: auto;">
</p>
## Fine-tuned Checkpoints
The following checkpoints store learnable parameters (positional embedding layers, Time-aware Frame Encoder, Sliding Video Q-Former, linear projection layers, and lora) only.
| Checkpoint | LLM backbone | Link | Note Excerpt of 9,707 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:8ce40e9471d8073a, desc:multimodal