Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
This repo lists relevant papers summarized in our survey paper: A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models.
| Date | Stars |
|---|---|
| 2026-07-31 | 514 |
| 2026-08-06 | 514 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Awesome Prompting on Vision-Language Models
<img src="./assets/pvlm-mindmap.png" width="100%" height="100%">
## # :nerd_face: What is Prompting on Vision-Language Models?
Prompt engineering is a technique that involves augmenting a large pre-trained model with task-specific hints, known as prompts, to adapt the model to new tasks. This repo aims to provide **a comprehensive survey** of cutting-edge research in prompt engineering on **three** types of vision-language models (VLMs): **multimodal-to-text generation models** (*e.g.*, Flamingo), **image-text matching models** (*e.g.*, CLIP), and **text-to-image generation models** (*e.g.*, Stable Diffusion) (Fig. 1).
<img src="./assets/3-models.png">
<p align="center"> <i>Fig. 1: This work focuses on three main types of vision-language models.</i> </p>
### Reference
This repo lists relevant papers summarized in our survey:
**A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models.** *[Jindong Gu](https://jindonggu.github.io/), [Zhen Han](https://sites.google.com/view/zhenhan/home?authuser=0), [Shuo Chen](https://chenxshuo.github.io/), [Ahmad Beirami](https://sites.google.com/view/beirami), [Bailan He](https://scholar.google.com/citations?user=n5zUQtAAAAAJ&hl=en), [Gengyuan Zhang](https://scholar.google.com/citations?user=LN2tYr0AAAAJ&hl=en), [Ruotong Liao](https://scholar.google.com/citations?user=XFQv_oYAAAAJ&hl=en), [Yao Qin](https://cseweb.ucsd.edu/~yaq007/), [Volker Tresp](https://www.dbs.ifi.lmu.de/~tresp/), [Philip Torr](https://torrvision.com/index.html)*. Preprint 2023. [[pdf]](https://arxiv.org/abs/2307.12980)
If you find our paper and repo helpful to your research, please cite the following paper:
```latex
@article{gu2023survey,
title={A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models},
author={Gu, Jindong and Han, Zhen and Chen, Shuo, and Beirami, Ahmad and He, Bailan and Zhang, Gengyuan and Liao, Ruotong and Qin, Yao and Tresp, Volker and Torr, Philip}
journal={arXiv preprint arXiv:2307.12980},
year={2023}
}
```
## # :paperclips: Awesome Papers
- [Prompting Model in Multimodal-to-Text Generation](#prompting-model-in-multimodal-to-text-generation-eg-on-flamingo)
- [Prompting Model in Image-Text Matching](#prompting-model-in-image-text-matching-eg-on-clip)
- [Prompting Model in Text-to-Image Generation](#prompting-model-in-text-to-image-generation-eg-on-stable-diffusion)
### Prompting Models in Multimodal-to-Text Generation (*e.g.* on Flamingo)
There are two main types of fusion module approaches based on the integration of visual and textual modalities: **encoder-decoder as a multi-modal fusion module** and **decoder-only as a multi-modal fusion module**. Prompting methods can be divided into **two main categories** (Fig. 2) based on the readability of the templates: **hard prompt** and **soft prompt**. Hard prompt encompasses four subcategories: *task instruction, in-context learning,* *retrieval-based prompting, and chain-of-thought prompting*. Soft prompts are classified into two strategies: *prompt tuning* and *prefix token tuning*, based on whether they internally add new tokens to the model's architecture or simply append them to the input. this study primarily concentrates on prompt methods that avoid altering the base model.
<img src="./assets/chapt3_prompting_method.png">
<p align="center"> <i>Fig. 2 : Classification of prompting methods.</i> </p>
| Title | Venue | Year | Code if available | Comment |
| :----------------------------------------------------------- | ----------- | ---- | ------------------------------------------------------------ | ---------------------------------------------------- |
| [Unifying Vision-and-Language Tasks via Text Generation](https://arxiv.org/abs/2102.02779) | ICML | 2021 | [GithuExcerpt of 37,477 characters
Read on GitHub
31
13
4
1
1
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:a64470822aa8efb4, name:vision-language, desc:vision-language, name:vision language model
matched fp:a64470822aa8efb4, topic:foundation-models
matched fp:a64470822aa8efb4, topic:prompt-engineering, desc:prompt engineering