Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Code, Dataset, and Pretrained Models for Audio and Speech Large Language Model "Listen, Think, and Understand".
| Date | Stars |
|---|---|
| 2026-07-24 | 478 |
| 2026-07-25 | 478 |
| 2026-07-28 | 478 |
| 2026-07-30 | 478 |
| 2026-07-31 | 478 |
| 2026-08-06 | 478 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Listen, Think, and Understand
- [Introduction](#introduction)
- [Citation](#citation)
- [OpenAQA (LTU) and OpenASQA (LTU-AS) Dataset](#openaqa-ltu-and-openasqa-ltu-as-dataset)
* [OpenAQA (LTU)](#for-ltu-openaqa)
* [OpenASQA (LTU-AS)](#for-ltu-as-openasqa)
- [Set the Virtual Environment](#set-the-virtual-environment)
- [Inference ](#inference)
* [Option 1. Inference via HuggingFace Space (No Code Needed)](#option-1-inference-via-huggingface-space-no-code-needed)
* [Option 2. Inference with API (No GPU Needed)](#option-2-inference-with-api-no-gpu-needed)
* [Option 3. Local Inference](#option-3-local-inference)
- [Finetune LTU and LTU-AS](#finetune-ltu-and-ltu-as)
* [Finetune the LTU/LTU-AS Model with Toy Data](#finetune-the-ltultu-as-model-with-toy-data)
* [Finetune the LTU/LTU-AS Model with Your Own Data](#finetune-the-ltultu-as-model-with-your-own-data)
- [Reproduce LTU and LTU-AS Training](#reproduce-ltu-and-ltu-as-training)
- [Pretrained Models](#pretrained-models)
- [Important Code](#important-code)
- [Required Computational Resources](#required-computational-resources)
- [Mirror Links](#mirror-links)
---
## Introduction
<p align="center"><img src="https://github.com/YuanGongND/ltu/blob/main/ltu.png?raw=true" alt="Illustration of CAV-MAE." width="900"/></p>
This repository contains the official implementation (in PyTorch), pretrained checkpoints, and datasets of LTU and LTU-AS.
LTU and LTU-AS are the first generation of audio and speech large language model that bridges audio/speech perception with understanding.
They not only achieve SOTA on multiple closed-ended audio and speech tasks, but also can answer any open-ended question based on the given audio.
Please try the interactive demos to see how good they are!
[**[LTU Interactive Demo]**](https://huggingface.co/spaces/yuangongfdu/LTU)
[**[LTU-AS Interactive Demo]**](https://huggingface.co/spaces/yuangongfdu/ltu-2)
---
## Citation
**LTU (First Generation, Only Supports Audio):**
*LTU was accepted at ICLR 2024. See you in Vienna!*
**[[Paper]](https://openreview.net/pdf?id=nBZBPXdJlC)** **[[HuggingFace Space]](https://huggingface.co/spaces/yuangongfdu/LTU)** **[[ICLR Peer Review]](https://openreview.net/forum?id=nBZBPXdJlC)**
**Authors:** [Yuan Gong](https://yuangongnd.github.io/), [Hongyin Luo](https://luohongyin.github.io/), [Alexander H. Liu](https://alexander-h-liu.github.io/), [Leonid Karlinsky](https://mitibmwatsonailab.mit.edu/people/leonid-karlinsky/), and [James Glass](https://people.csail.mit.edu/jrg/) (MIT & MIT-IBM Watson AI Lab)
```
@article{gong2023listen,
title={Listen, Think, and Understand},
author={Gong, Yuan and Luo, Hongyin and Liu, Alexander H and Karlinsky, Leonid and Glass, James},
journal={arXiv preprint arXiv:2305.10790},
year={2023}
}
```
---
**LTU-AS (Second Generation, Supports Speech and Audio):**
*LTU-AS was accepted at ASRU 2023 (top 3% paper). See you in Taipei!*
**[[Paper]](https://arxiv.org/pdf/2309.14405.pdf)** **[[HuggingFace Space]](https://huggingface.co/spaces/yuangongfdu/ltu-2)** **[[ASRU Peer Review]](https://github.com/YuanGongND/ltu/tree/main/asru_review)**
**Authors:** [Yuan Gong](https://yuangongnd.github.io/), [Alexander H. Liu](https://alexander-h-liu.github.io/), [Hongyin Luo](https://luohongyin.github.io/), [Leonid Karlinsky](https://mitibmwatsonailab.mit.edu/people/leonid-karlinsky/), and [James Glass](https://people.csail.mit.edu/jrg/) (MIT & MIT-IBM Watson AI Lab)
```
@inproceedings{gong_ltuas,
title={Joint Audio and Speech Understanding},
author={Gong, Yuan and Liu, Alexander H and Luo, Hongyin, and Karlinsky, Leonid and Glass, James},
year={2023},
booktitle={2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)},
}
```
---
## OpenAQA (LTU) and OpenASQA (LTU-AS) Dataset
We release the training data for LTU (OpenAQA) and LTU-AS (OpenASQA). Specifically, we release the (`question`, `answer`, `audio_id`) tuples.
The actual audio files areExcerpt of 26,702 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:f2939482f4774701, topic:speech-recognition, topic:audio-processing, readme:speech recognition
matched fp:f2939482f4774701, topic:deep-learning
matched fp:f2939482f4774701, topic:large-language-models