Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Vision-language models, document understanding and any-to-any architectures.
Signals: multimodal, vision-language-model, vlm, clip, multimodal-llm, visual-question-answering, document-understanding, image-captioning
344 results
This repository contains various models targetting multimodal representation learning, multimodal fusion for downstream tasks such as multimodal sentiment analysis.
FaceChain is a deep-learning toolchain for generating your Digital-Twin.
🧠 Web Neural Network API
NeuralTalk is a Python+numpy project for learning Multimodal Recurrent Neural Networks that describe images with sentences.
MM-EUREKA: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning
Caption-Anything is a versatile tool combining image segmentation, visual captioning, and ChatGPT, generating tailored captions with diverse controls for user preferences. https://huggingface.co/spaces/TencentARC/Caption-Anything https://huggingface.co/spaces/VIPLab/Caption-Anything
The simplest, fastest repository for training/finetuning small-sized VLMs.
No description
[CVPR 2025] Magma: A Foundation Model for Multimodal AI Agents
Give Claude the ability to watch any video. /watch downloads, extracts frames, transcribes, hands it all to Claude.
Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey
An official implementation for "CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval"
[ICLR'24 spotlight] Chinese and English Multimodal Large Model Series (Chat and Paint) | 基于CPM基础模型的中英双语多模态大模型系列
🩺 首个会看胸部X光片的中文多模态医学大模型 | The first Chinese Medical Multimodal Model that Chest Radiographs Summarization.
Codebase for Aria - an Open Multimodal Native MoE
Implementation of CoCa, Contrastive Captioners are Image-Text Foundation Models, in Pytorch
A curated list of Multimodal Related Research.
[ICLR & NeurIPS 2025] Repository for Show-o series, One Single Transformer to Unify Multimodal Understanding and Generation.
[ECCV2024] Video Foundation Models & Data for Multimodal Understanding
mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding
Open-source multimodal retrieval engine (Morphik Core). By Morphik — AI back office for skilled nursing & senior living (morphik.ai).
No description
Easily compute clip embeddings and build a clip retrieval system with them
A modular library built on top of Keras and TensorFlow to generate a caption in natural language for any input image.
24,523 repositories in the index in total.
