Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Train Models Contrastively in Pytorch
| Date | Stars |
|---|---|
| 2026-07-24 | 798 |
| 2026-07-25 | 799 |
| 2026-07-28 | 799 |
| 2026-07-30 | 799 |
| 2026-08-06 | 799 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# contrastors `contrastors` is contrastive learning toolkit that enables researchers and engineers to train and evaluate contrastive models efficiently. [](https://atlas.nomic.ai/map/nomic-text-embed-v1-5m-sample) ## Features - Built on top of [Flash Attention](https://github.com/Dao-AILab/flash-attention) for fast and efficient training - Support for training on multiple GPUs - [GradCache](https://github.com/luyug/GradCache) support for training with large batch sizes in constrained memory environments - Huggingface Support for easy loading of common models (Pythia/GPTNeoX, BERT, etc.) - Masked Language Modeling (MLM) Pretraining - [Matryoshka Representation Learning](https://arxiv.org/abs/2205.13147) for flexible embedding sizes - [CLIP](https://arxiv.org/abs/2103.00020) and [LiT](https://arxiv.org/abs/2111.07991) style contrastive learning - Support for loading popular ViT (e.g. [timm](https://huggingface.co/timm)) models ## Research * [Nomic Embed: Training a Reproducible Long Context Text Embedder](https://arxiv.org/abs/2402.01613) by Zach Nussbaum, Jack Morris, Andriy Mulyar, and Brandon Duderstadt * [Nomic Embed Vision: Expanding the Latent Space](https://arxiv.org/abs/2406.18587) by Zach Nussbaum, Brandon Duderstadt, and Andriy Mulyar * [Training Sparse Mixture Of Experts Text Embedding Models](https://static.nomic.ai/nomic_embed_multi_preprint.pdf) by Zach Nussbaum and Brandon Duderstadt ## Getting Started and Requirements The `contrastors` library relies on custom kernels from the [Flash Attention](https://github.com/Dao-AILab/flash-attention) repository. To setup your enviornment you will need to follow the steps below. Make sure that you have Cuda 11.8+. You can check this by running `nvcc --version` or if you already have torch installed you can run `python -c "import torch; print(torch.version.cuda)"` Create a python venv and activate it ```bash python3 -m venv env source env/bin/activate ``` Install [torch](https://pytorch.org/get-started/locally/). See the torch docs for specific instructions for your system (e.g. the default CUDA torch supports is 12.1 as of 12/12/2023). ```bash pip3 install torch torchvision torchaudio ``` Install wheel, packaging, ninja for Flash Attention (so the builds don't take too long) ```bash pip install wheel packaging ninja setuptools ``` Install Flash Attention and the custom kernels ```bash pip install --no-cache-dir flash-attn --no-build-isolation git+https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary git+https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/layer_norm git+https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/fused_dense_lib git+https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/xentropy ``` Install the rest of the requirements and the package ```bash pip install -e . ``` ## Data Access We provide access to the `nomic-embed-text-v1` dataset via the `nomic` package. To access the data, you will need to create an account and login to the `nomic` package. First create an account at [atlas.nomic.ai](https://atlas.nomic.ai), download the `nomic` Python client, and run the following commands: ```bash pip install nomic nomic login # follow prompts to login python -c "from nomic import atlas; print(atlas._get_datastream_credentials(name='contrastors'))" ``` which will print out your access keys. You can then configure them by using `aws configure` or setting the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. If you do not have the AWS CLI installed, you can install it [here](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). To verify your access, you can run the following command to list the contents of the bucket: ```bash aws s3 ls --endpoint-url=https://9fa58365a1a3d032127970d0bd9a1290.r2.cloudflarestorage.com/ s3://contrastive aws s3 ls --endpoint-url=https://9fa58365a1a3d0321
Excerpt of 9,777 characters
Read on GitHub
Zach Nussbaum
32
5
2
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:29293c1b4c094a98, topic:embeddings, topic:text-embeddings, topic:contrastive-learning
matched fp:29293c1b4c094a98, topic:deep-learning, topic:pytorch, readme:pretraining