Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Official Pytorch implementation of "OmniNet: A unified architecture for multi-modal multi-task learning" | Authors: Subhojeet Pramanik, Priyanka Agrawal, Aman Hussain
| Date | Stars |
|---|---|
| 2026-07-24 | 514 |
| 2026-07-25 | 514 |
| 2026-07-28 | 514 |
| 2026-07-30 | 514 |
| 2026-08-06 | 514 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# OmniNet: A unified architecture for multi-modal multi-task learning [](https://opensource.org/licenses/Apache-2.0) *OmniNet* is a unified and extended version of the Transformer architecture for multi-modal multi-task learning. A single OmniNet architecture can encode multiple inputs from almost any real-life domain (text, image, video) and is capable of asynchronous multi-task learning across a wide range of tasks. The *OmniNet* architecture consists multiple sub-networks called the neural peripherals, used to encode domain specific inputs as spatio-temporal representations, connected to a common central neural network called the Central Neural Processor (CNP). The CNP implements a Transformer based universal spatio-temporal encoder and a multi-task decoder. This repository contains the official Pytorch implementation for <a href="https://arxiv.org/abs/1907.07804">OmniNet: A unified architecture for multi-modal multi-task learning</a> (Pramanik et al). The paper demonstrates a single instance of *OmniNet* jointly trained to perform the tasks of part-of-speech tagging, image captioning, visual question answering and video activity recognition. We also open source the pre-trained models to be able to reproduce the results demonstrated in the paper. <p align="center"> <img src="resources/omninet_architecture.png" width="700"> </p> ## Setup **Requirements** - Minimum hardware: 8GB RAM + NVIDIA GPU (8GB+) - Linux based OS - NVIDIA driver 410+ - [Anaconda Package Manager](https://anaconda.org/) - rest will be installed by anaconda (see below) **Installation** 1. Clone the repo. 2. Create & activate an environment using the provided file. (This will install all required dependencies including cuda and cudnn) ``` $ conda env create -f environment.yml $ source activate omninet ``` **Download datasets** A download script has been provided to download and preprocess all training and test data for all tasks (except PENN POS tagging) mentioned in the paper. The downloaded data is stored in /data folder. ``` $ python scripts/init_setup.py ``` ## Pretrained models Pretrained models for the various single and multi-task models demonstrated in the paper can be obtained from the following URLs and can be used to evaluate the performance of the architecture or perform prediction on external data. 1. VQA: https://storage.googleapis.com/omninet/200k_vqa.zip 2. Captioning: https://storage.googleapis.com/omninet/280k_caption.zip 3. HMDB: https://storage.googleapis.com/omninet/11k_hmdb.zip 4. PENN: https://storage.googleapis.com/omninet/70k_penn.zip 5. VQA+HMDB+Captioning+PENN: https://storage.googleapis.com/omninet/all_15k.zip ## Training The training script includes subroutines to train the model on the COCO Image captioning, VQA, HMDB action recognition, and PENN POS tagging. First, download all required datasets using `scripts/init_setup.py` as described above. The training script can be used to train both on a single task and asynchronously on multiple different tasks. It implements multi-GPU hogwild training to train a single model across multiple GPUs on multiple tasks. ``` $ python train.py <no. of training iterations> <task list> <batch sizes> --n_gpus <no. of GPUs> \ --save_interval <save interval> --eval_interval <eval interval> ``` The list of possible supported tasks are `vqa`, `hmdb`, `caption` and `penn`. (The data downloader in init_setup.py, however, does not download datasets for PENN tagging and it is required to manually download and preprocess PENN POS tagging dataset to train the model for POS tagging.) For example, to train the model on a single task of Visual Question Answering for 100000 iterations and batch size of 128, use: ``` $ python train.py 100000 vqa 128 --n_gpus 1 --save_interval 500 --eval_interval 500 ``` To train the model asynchronously on multiple tasks specify the list of tasks and batch sizes separated by a comma. Fo
Excerpt of 9,068 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:9974d77a282afced, topic:deep-learning, topic:neural-network
matched fp:9974d77a282afced, topic:transformer, readme:transformer architecture
matched fp:9974d77a282afced, topic:image-captioning, readme:visual question answering, readme:image captioning
matched fp:9974d77a282afced, topic:nlp, readme:part-of-speech
