Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Bottom-up attention model for image captioning and VQA, based on Faster R-CNN and Visual Genome
| Date | Stars |
|---|---|
| 2026-07-24 | 1470 |
| 2026-07-25 | 1470 |
| 2026-07-28 | 1470 |
| 2026-07-30 | 1470 |
| 2026-08-06 | 1470 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# bottom-up-attention
This code implements a bottom-up attention model, based on multi-gpu training of Faster R-CNN with ResNet-101, using object and attribute annotations from [Visual Genome](http://visualgenome.org/).
The pretrained model generates output features corresponding to salient image regions. These bottom-up attention features can typically be used as a drop-in replacement for CNN features in attention-based image captioning and visual question answering (VQA) models. This approach was used to achieve state-of-the-art image captioning performance on [MSCOCO](https://competitions.codalab.org/competitions/3221#results) (**CIDEr 117.9**, **BLEU_4 36.9**) and to win the [2017 VQA Challenge](http://www.visualqa.org/workshop.html) (**70.3%** overall accuracy), as described in:
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://arxiv.org/abs/1707.07998), and
- [Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge](https://arxiv.org/abs/1708.02711).
Some example object and attribute predictions for salient image regions are illustrated below.
Note: This repo only includes code for training the bottom-up attention / Faster R-CNN model (section 3.1 of the [paper](https://arxiv.org/abs/1707.07998)). The actual captioning model (section 3.2) is available in a separate repo [here](https://github.com/peteanderson80/Up-Down-Captioner).
### Reference
If you use our code or features, please cite our paper:
```
@inproceedings{Anderson2017up-down,
author = {Peter Anderson and Xiaodong He and Chris Buehler and Damien Teney and Mark Johnson and Stephen Gould and Lei Zhang},
title = {Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering},
booktitle={CVPR},
year = {2018}
}
```
### Disclaimer
This code is modified from [py-R-FCN-multiGPU](https://github.com/bharatsingh430/py-R-FCN-multiGPU), which is in turn modified from [py-faster-rcnn](https://github.com/rbgirshick/py-faster-rcnn) code. Please refer to these links for further README information (for example, relating to other models and datasets included in the repo) and appropriate citations for these works. This README only relates to Faster R-CNN trained on Visual Genome.
### License
bottom-up-attention is released under the MIT License (refer to the LICENSE file for details).
### Pretrained features
For ease-of-use, we make pretrained features available for the entire [MSCOCO dataset](http://mscoco.org/dataset/#download). It is not necessary to clone or build this repo to use features downloaded from the links below. Features are stored in tsv (tab-separated-values) format that can be read with `tools/read_tsv.py`.
**LINKS HAVE BEEN UPDATED TO GOOGLE CLOUD STORAGE (14 Feb 2021)**
10 to 100 features per image (adaptive):
- [2014 Train/Val Image Features (120K / 23GB)](https://storage.googleapis.com/up-down-attention/trainval.zip)
- [2014 Testing Image Features (40K / 7.3GB)](https://storage.googleapis.com/up-down-attention/test2014.zip)
- [2015 Testing Image Features (80K / 15GB)](https://storage.googleapis.com/up-down-attention/test2015.zip)
36 features per image (fixed):
- [2014 Train/Val Image Features (120K / 25GB)](https://storage.googleapis.com/up-down-attention/trainval_36.zip)
- [2014 Testing Image Features (40K / 9GB)](https://storage.googleapis.com/up-down-attention/test2014_36.zip)
- [2015 Testing Image Features (80K / 17GB)](https://storage.googleapis.com/up-down-attention/test2015_36.zip)
Both sets of features can be recreated by using `tools/generate_tsv.py` with the appropriate pretrained model and with MIN_BOXES/MAX_BOXES set to either 10/100 or 36/36 respectively - refer [Demo](#demo).
### Contents
1. [Requirements: software](#requirements-software)
2. [Requirements: hardware](#requirements-hardware)
3. [Basic installation](#instaExcerpt of 9,552 characters
Read on GitHub
28
16
6
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:88bc01dd8f04d6b5, topic:visual-question-answering, topic:image-captioning, readme:visual question answering