Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Implementation of Enformer, Deepmind's attention network for predicting gene expression, in Pytorch
| Date | Stars |
|---|---|
| 2026-07-24 | 573 |
| 2026-07-25 | 572 |
| 2026-07-28 | 572 |
| 2026-07-30 | 572 |
| 2026-08-06 | 572 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<img src="./enformer.png" width="450px"></img>
## Enformer - Pytorch
Implementation of <a href="https://deepmind.com/blog/article/enformer">Enformer</a>, Deepmind's attention network for predicting gene expression, in Pytorch. This repository also contains the means to fine tune pretrained models for your downstream tasks. The original tensorflow sonnet code can be found <a href="https://github.com/deepmind/deepmind-research/tree/master/enformer">here</a>.
Update: finetuned for predicting pseudobulk chromatin accessibility <a href="https://www.biorxiv.org/content/10.1101/2023.11.27.568764v1">here</a>
## Install
```bash
$ pip install enformer-pytorch
```
## Usage
```python
import torch
from enformer_pytorch import Enformer
model = Enformer.from_hparams(
dim = 1536,
depth = 11,
heads = 8,
output_heads = dict(human = 5313, mouse = 1643),
target_length = 896,
)
seq = torch.randint(0, 5, (1, 196_608)) # for ACGTN, in that order (-1 for padding)
output = model(seq)
output['human'] # (1, 896, 5313)
output['mouse'] # (1, 896, 1643)
```
You can also directly pass in the sequence as one-hot encodings, which must be float values
```python
import torch
from enformer_pytorch import Enformer, seq_indices_to_one_hot
model = Enformer.from_hparams(
dim = 1536,
depth = 11,
heads = 8,
output_heads = dict(human = 5313, mouse = 1643),
target_length = 896,
)
seq = torch.randint(0, 5, (1, 196_608))
one_hot = seq_indices_to_one_hot(seq)
output = model(one_hot)
output['human'] # (1, 896, 5313)
output['mouse'] # (1, 896, 1643)
```
Finally, one can fetch the embeddings, for fine-tuning and otherwise, by setting the `return_embeddings` flag to be `True` on forward
```python
import torch
from enformer_pytorch import Enformer, seq_indices_to_one_hot
model = Enformer.from_hparams(
dim = 1536,
depth = 11,
heads = 8,
output_heads = dict(human = 5313, mouse = 1643),
target_length = 896,
)
seq = torch.randint(0, 5, (1, 196_608))
one_hot = seq_indices_to_one_hot(seq)
output, embeddings = model(one_hot, return_embeddings = True)
embeddings # (1, 896, 3072)
```
For training, you can directly pass the head and target in to get the poisson loss
```python
import torch
from enformer_pytorch import Enformer, seq_indices_to_one_hot
model = Enformer.from_hparams(
dim = 1536,
depth = 11,
heads = 8,
output_heads = dict(human = 5313, mouse = 1643),
target_length = 200,
).cuda()
seq = torch.randint(0, 5, (196_608 // 2,)).cuda()
target = torch.randn(200, 5313).cuda()
loss = model(
seq,
head = 'human',
target = target
)
loss.backward()
# after much training
corr_coef = model(
seq,
head = 'human',
target = target,
return_corr_coef = True
)
corr_coef # pearson R, used as a metric in the paper
```
## Pretrained Model
Deepmind has released the weights for their tensorflow sonnet Enformer model! I have ported it over to Pytorch and uploaded it to <a href="https://huggingface.co/EleutherAI/enformer-official-rough">🤗 Huggingface</a> (~1GB). There are still some rounding errors that seem to be accruing across the layers, resulting in an absolute error as high as `0.5`. However, correlation coefficient look good so I am releasing the 'rough'ly working version. Will keep working on figuring out where the numerical errors are happening (it may be the attention pooling module, as I noticed the attention logits are pretty high).
Update: <a href="https://github.com/jstjohn">John St. John</a> did some work and found that the `enformer-official-rough` model hits the reported marks in the paper - human pearson R of `0.625` for validation, and `0.65` for test.
Update: As of version 0.8.0, if one were to use the `from_pretrained` function to load the pretrained model, it should automatically use precomputed gamma positions to address a difference between tensorflow and pytorch `xlogy`. This should resolve the numerical discrepancy above. If you were tExcerpt of 13,078 characters
Read on GitHub
Phil Wang · United States
166
2
Julien Chaumond · @huggingface · United States
1
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:719d90b631791d77, topic:transformer, readme:pretrained model
matched fp:719d90b631791d77, topic:deep-learning