Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Code for 'LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders'
| Date | Stars |
|---|---|
| 2026-07-31 | 1707 |
| 2026-08-03 | 1708 |
| 2026-08-06 | 1708 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# *LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders*
[](https://arxiv.org/abs/2404.05961)
[](https://pypi.org/project/llm2vec/)
[](https://huggingface.co/collections/McGill-NLP/llm2vec-660e14f536b3c8d10b3f1c34)
[](https://github.com/McGill-NLP/llm2vec/blob/main/LICENSE)
[](https://pepy.tech/project/llm2vec)
LLM2Vec is a simple recipe to convert decoder-only LLMs into text encoders. It consists of 3 simple steps: 1) enabling bidirectional attention, 2) training with masked next token prediction, and 3) unsupervised contrastive learning. The model can be further fine-tuned to achieve state-of-the-art performance.
<p align="center">
<img src="https://github.com/McGill-NLP/llm2vec/assets/12207571/48efd48a-431b-4625-8e0f-248a442e3839" width="75%" alt="LLM2Vec_figure1"/>
</p>
**************************** **Updates** ****************************
* 04/04/26: We released LLM2Vec-Gen🚀, a recipe to train interpretable, generative embeddings that encode the potential answer of an LLM to a query rather than the query itself! Please checkout our 📄[paper](https://arxiv.org/abs/2603.10913), 💻[GitHub repository](https://github.com/McGill-NLP/llm2vec-gen),and 🤗[HF models](https://huggingface.co/McGill-NLP/LLM2Vec-Gen-Qwen3-8B) for more information.
* 03/10: Added support for latest transformer versions, which support Llama 3.1, 3.2 and other latest models. Expanded support to evaluate any LLM2vec model, check [mteb_eval_custom.py](https://github.com/McGill-NLP/llm2vec/blob/main/experiments/mteb_eval_custom.py)
* 04/07: Added support for Gemma and Qwen-2 models, huge thanks to [@bzantium](https://github.com/bzantium) for the contribution.
* 30/04: We release LLM2Vec transformed Meta-Llama-3 checkpoints. See our [HuggingFace collection](https://huggingface.co/collections/McGill-NLP/llm2vec-660e14f536b3c8d10b3f1c34) for both [supervised](https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised) and [unsupervised](https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-unsup-simcse) variants.
## Installation
To use LLM2Vec, first install the llm2vec package from PyPI, followed by installing flash-attention:
```bash
pip install llm2vec
pip install flash-attn --no-build-isolation
```
You can also directly install the latest version of llm2vec by cloning the repository:
```bash
pip install -e .
pip install flash-attn --no-build-isolation
```
## Getting Started
LLM2Vec class is a wrapper on top of HuggingFace models to support enabling bidirectionality in decoder-only LLMs, sequence encoding and pooling operations. The steps below showcase an example on how to use the library.
### Preparing the model
Initializing LLM2Vec model using pretrained LLMs is straightforward. The `from_pretrained` method of LLM2Vec takes a base model identifier/path and an optional PEFT model identifier/path. All HuggingFace model loading arguments can be passed to `from_pretrained` method. By default, the models are loaded with bidirectional connections enabled. This can be turned off by passing `enable_bidirectional=False` to the `from_pretrained` method.
Here, we first initialize the Llama-3 MNTP base model and load the unsupervised-trained LoRA weights (trained with SimCSE objective and wiki corpus).
```python
import torch
from llm2vec import LLM2Vec
l2v = LLM2Vec.from_pretrained(
"McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
peft_model_name_or_path="McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-unsup-simcse",
device_map="cuda" if torch.cuda.is_available() else "cpu",
torch_dtype=torch.bfloat16,
)
```
We can also load the model with supervised-trained LoRA weights (Excerpt of 16,034 characters
Read on GitHub
130
30
2
1
1
1
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:a05936cbee7f48f2, llm:Repository title and description: 'LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders' — code for paper demonstrating LLMs as text encoders.
matched fp:a05936cbee7f48f2, llm:Repository title and description: 'LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders' — code for paper demonstrating LLMs as text encoders.