Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Visual Speech Recognition for Multiple Languages
| Date | Stars |
|---|---|
| 2026-07-31 | 479 |
| 2026-08-01 | 479 |
| 2026-08-06 | 479 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<p align="center"><img width="160" src="doc/lip_white.png" alt="logo"></p>
<h1 align="center">Visual Speech Recognition for Multiple Languages</h1>
<div align="center">
[📘Introduction](#Introduction) |
[🛠️Preparation](#Preparation) |
[📊Benchmark](#Benchmark-evaluation) |
[🔮Inference](#Speech-prediction) |
[🐯Model zoo](#Model-Zoo) |
[📝License](#License)
</div>
## Authors
[Pingchuan Ma](https://mpc001.github.io/), [Alexandros Haliassos](https://dblp.org/pid/257/3052.html), [Adriana Fernandez-Lopez](https://scholar.google.com/citations?user=DiVeQHkAAAAJ), [Honglie Chen](https://scholar.google.com/citations?user=HPwdvwEAAAAJ), [Stavros Petridis](https://ibug.doc.ic.ac.uk/people/spetridis), [Maja Pantic](https://ibug.doc.ic.ac.uk/people/mpantic).
## Update
`2023-07-26`: We have released our training recipe for real-time AV-ASR, see [here](https://github.com/pytorch/audio/tree/main/examples/avsr).
`2023-06-16`: We have released our training recipe for AutoAVSR, see [here](https://github.com/mpc001/auto_avsr).
`2023-03-27`: We have released our AutoAVSR models for LRS3, see [here](#autoavsr-models).
## Introduction
This is the repository of [Visual Speech Recognition for Multiple Languages](https://arxiv.org/abs/2202.13084), which is the successor of [End-to-End Audio-Visual Speech Recognition with Conformers](https://arxiv.org/abs/2102.06657). By using this repository, you can achieve the performance of 19.1%, 1.0% and 0.9% WER for automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR) on LRS3.
## Tutorial
We provide a tutorial [](https://colab.research.google.com/drive/1jfb6e4xxhXHbmQf-nncdLno1u0b4j614) to show how to use our Auto-AVSR models to perform speech recognition (ASR, VSR, and AV-ASR), crop mouth ROIs or extract visual speech features.
## Demo
English -> Mandarin -> Spanish | French -> Portuguese -> Italian |
:-------------------------------:|:------------------------------------:
<img src='doc/vsr_1.gif' title='vsr1' style='max-width:320px'></img> | <img src='doc/vsr_2.gif' title='vsr2' style='max-width:320px'></img> |
<div align="center">
[Youtube](https://youtu.be/FIau-6JA9Po) |
[Bilibili](https://www.bilibili.com/video/BV1Wu411D7oP)
</div>
## Preparation
1. Clone the repository and enter it locally:
```Shell
git clone https://github.com/mpc001/Visual_Speech_Recognition_for_Multiple_Languages
cd Visual_Speech_Recognition_for_Multiple_Languages
```
2. Setup the environment.
```Shell
conda create -y -n autoavsr python=3.8
conda activate autoavsr
```
3. Install pytorch, torchvision, and torchaudio by following instructions [here](https://pytorch.org/get-started/), and install all packages:
```Shell
pip install -r requirements.txt
conda install -c conda-forge ffmpeg
```
4. Download and extract a pre-trained model and/or language model from [model zoo](#Model-Zoo) to:
- `./benchmarks/${dataset}/models`
- `./benchmarks/${dataset}/language_models`
5. [For VSR and AV-ASR] Install [RetinaFace](./tools) or [MediaPipe](https://pypi.org/project/mediapipe/) tracker.
### Benchmark evaluation
```Shell
python eval.py config_filename=[config_filename] \
labels_filename=[labels_filename] \
data_dir=[data_dir] \
landmarks_dir=[landmarks_dir]
```
- `[config_filename]` is the model configuration path, located in `./configs`.
- `[labels_filename]` is the labels path, located in `${lipreading_root}/benchmarks/${dataset}/labels`.
- `[data_dir]` and `[landmarks_dir]` are the directories for original dataset and corresponding landmarks.
- `gpu_idx=-1` can be added to switch from `cuda:0` to `cpu`.
### Speech prediction
```Shell
python infer.py config_filename=[config_filename] data_filename=[data_filename]
```
- `data_filename` is the path to the audio/video file.
- `detector=mediapipe` can be added to switch from RetinaFace to MediaPipe tracker.
### MoutExcerpt of 14,117 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:be06074668886ba6, name:speech recognition, desc:speech recognition