Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Theia: Distilling Diverse Vision Foundation Models for Robot Learning
| Date | Stars |
|---|---|
| 2026-07-31 | 277 |
| 2026-08-06 | 277 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<h1 align="center">Theia: Distilling Diverse Vision Foundation Models for Robot Learning</h1>
<h4 align="center">
<a href="https://www3.cs.stonybrook.edu/~jishang" target="_blank">Jinghuan Shang</a><sup>1,2</sup>, <a href="https://sites.google.com/view/karlschmeckpeper" target="_blank">Karl Schmeckpeper</a><sup>1</sup>, <a href="https://scholar.google.com/citations?user=_UnlC7IAAAAJ&hl=en" target="_blank">Brandon B. May</a><sup>1</sup>, Maria Vittoria Minniti<sup>1</sup>, <a href="http://kelestemur.com" target="_blank">Tarik Kelestemur</a><sup>1</sup>, <a href="https://davidjosephwatkins.com" target="_blank">David Watkins</a><sup>1</sup>, Laura Herlant<sup>1</sup>
</h4>
<p align="center">
<sup>1</sup><a href="http://theaiinstitute.com/" target="_blank">The AI Institute</a>
<sup>2</sup><a href="https://www3.cs.stonybrook.edu/~cvl/" target="_blank">Stony Brook University</a>
</p>
<p align="center">
<b>CoRL 2024</b>
</p>
<p align="center">
<a href="http://theia.theaiinstitute.com/" target="_blank">Project Page</a>, <a href="https://openreview.net/forum?id=ylZHvlwUcI" target="_blank">Paper</a>, <a href="https://huggingface.co/collections/theaiinstitute/theia-66a7a6ae80a707547c358cce" target="_blank">Models</a>, <a href="https://huggingface.co/spaces/theaiinstitute/theia" target="_blank">Demo</a>
</p>
<p align="center">
<!-- <img src="doc/theia_overview.gif"> -->
<img src="doc/theia_overview.gif" height="300px">
</p>
## Quick Start: Use Pre-trained Theia Models
Through huggingface:
```
import transformers
from transformers import AutoModel
import torch
model = AutoModel.from_pretrained("theaiinstitute/theia-base-patch16-224-cdiv", trust_remote_code=True)
fake_input = torch.zeros((1, 224 ,224, 3), dtype=torch.uint8)
theia_feature = model.forward_feature(fake_input)
# Theia / intermediate feature, mainly used for robot learning.
# To change different feature reduction methods, pass `feature_reduction_method` argument in AutoModel.from_pretrained() method
predicted_features = model(fake_input)
# predicted_features is dict[str, torch.Tensor] where each kv pair is target model name and predicted feature
# they are predicted features that tries to match teacher model features.
```
`theia-<size>-patch16-224-cdiv` are used for main evaluations in the paper.
## Installation
Make sure you have Python >= 3.10. Create any virtual Python environment you like or use the [Dockerfile](./Dockerfile). Then
```
pip install -e .
```
## Data Preparation
### Datasets
The datasets should be organized in webdataset format.
1. Prepare images from ImageNet
First download and [prepare](https://gist.github.com/antoinebrl/7d00d5cb6c95ef194c737392ef7e476a) ImageNet.
```
cd src/theia/scripts/preprocessing/image_datasets
python organize_imagenet_webdataset.py --dataset <dataset_name> --imagenet-raw-path <path_to_raw_images> --output-path <root_dir_to_hold_datasets>
```
For any other image dataset you want to use, you can simply dump all of them in a folder (any subfolder also works), and modify how you can get their paths in `organize_imagenet_webdataset.py` (variable `image_paths`).
2. (Optional) Prepare frames from video datasets
```
cd src/theia/scripts/preprocessing/video_datasets
python subsampling_videos.py --dataset <dataset_name> --dataset-path <path_to_raw_videos> --output-path <root_dir_to_hold_datasets> [--subsampling-rate] [--samples-per-shard]
```
### Feature Extraction
```
cd src/theia/scripts/preprocessing
python feature_extraction.py --dataset <dataset_name> --output-path <root_dir_to_hold_datasets> --model <model_name> --split <train or val (or test)> [--num-gpus]
```
You can also refer to the integrated script `src/theia/scripts/preprocessing/iv_feature_extraction.py` that launches feature extraction for multiple models at the same time.
During training we will need mean and variance for each teacher model to normalize teacher features. You can extract them using `src/theia/scripts/preprocessing/calc_feature_mean.py` or uExcerpt of 7,199 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:f8a71d7987efc315, desc:robot learning