Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[NeurIPS 2023] Official implementations of "Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models"
| Date | Stars |
|---|---|
| 2026-07-31 | 523 |
| 2026-08-06 | 523 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
 --- This repository contains the implementation of the NeurIPS 2023 paper: > **Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models** > [[Project Page]](https://luogen1996.github.io/lavin/) [[Paper]](https://arxiv.org/pdf/2305.15023.pdf) <br> > [Gen Luo](https://luogen1996.github.io)<sup>1</sup>, Yiyi Zhou<sup>12</sup>, [Tianhe Ren](https://rentainhe.github.io)<sup>1</sup>, Shengxin Chen<sup>1</sup>, [Xiaoshuai Sun](https://sites.google.com/view/xssun)<sup>12</sup>, [Rongrong Ji](https://mac.xmu.edu.cn/rrji/)<sup>12</sup><br> <sup>1</sup>Media Analytics and Computing Lab, Department of Artificial Intelligence, School of Informatics, Xiamen University > <sup>2</sup>Institute of Artificial Intelligence, Xiamen University In this work, we propose a novel and affordable solution for vision-language instruction tuning, namely Mixture-of-Modality Adaptation (MMA). Particularly, MMA is an end-to-end optimization regime, which connects the image encoder and LLM via lightweight adapters. Meanwhile, we also propose a novel routing algorithm in MMA, which can help the model automatically shifts the reasoning paths for single- and multi-modal instructions. Based on MMA, we develop a large vision-language instructed model called LaVIN, which demonstrates superior training efficiency and better reasoning ability than existing multimodal LLMs in various instruction-following tasks. --- <div align="center"> <img src="./assets/teaser-1.png" width="95%"> </div> ## News - **`2023/09/22`**: 🔥🔥🔥 Our paper is accepted by NeurIPS 2023! - **`2023/06/30`**: 🔥🔥🔥 With very limited training data and cost, LaVIN achieves 5-th place of Perception and Cognition on [MME benchmark](https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation)， outperforming seven existing multimodal LLMs. Evaluation codes are available. - **`2023/06/27`**: 🔥4-bit trainings are available now ! LaVIN-lite can be trained on one 3090 GPU, taking around 9G and 15G GPU memory for the scales of 7B and 13B , respectively. Technical details are available in [知乎](https://zhuanlan.zhihu.com/p/638784025). - **`2023/05/29`**: 🔥We released the demo and the pre-trained checkpoint (LLaMA-13B) for multimodal chatbot. - **`2023/05/25`**: 🔥We released the code of **LaVIN: Large Vision-Language Instructed model**, which achieves 89.4 (LaVIN-7B) and 90.8 (LaVIN-13B) accuracy on ScienceQA! 🔥With the proposed **mixture-of-modality adaptation**, the training time and trainable parameters can be reduced to 1.4 hours and 3.8M, respectively! Checkout the [paper](https://arxiv.org/pdf/2305.15023.pdf). ## TODO - [x] Release training codes. - [x] Release checkpoints and demo. - [x] 4-bit training. - [ ] Support more modalities, e.g., audio and video. ## Contents - [Setup](#setup) - [Fine-tuning](#fine-tuning) - [Demo](#demo) - [Model Zoo](#model-zoo) ## Setup ### Install Package ```bash conda create -n lavin python=3.8 -y conda activate lavin # install pytorch conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 -c pytorch # install dependency and lavin pip install -r requirements.txt pip install -e . ``` ### Data Preparation - For ScienceQA, please prepare the dataset from the [official repo](https://github.com/lupantech/ScienceQA). - For Multimodal Chatbot, download the images in _train2014_ split from [MSCOCO](http://images.cocodataset.org/zips/train2014.zip), and obtain the prepared 52k text-only and 158k text-image instruction-following data from [here](https://drive.google.com/file/d/1gORDPruqwXbgy6NYmhpDXO7t089yzsg3/view?usp=share_link). - Obtain the weights of LLaMA from [this form](https://forms.gle/jk851eBVbX1m5TAv5) (official) or Download [LLaMA-7B](https://huggingface.co/nyanko7/LLaMA-7B/tree/main) and [LLaMA-13B](https://huggingface.co/TheBloke/llama-13b) from HuggingFace (unofficial). - If you want to use Vicuna weights to initialize the m
Excerpt of 9,960 characters
Read on GitHub
62
Ren Tianhe · IDEA · China
4
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:80fe458a8eedf4e1, desc:instruction tuning