Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
The official implementation for ICLR23 spotlight paper "DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion"
| Date | Stars |
|---|---|
| 2026-07-24 | 313 |
| 2026-07-25 | 313 |
| 2026-07-28 | 313 |
| 2026-07-30 | 313 |
| 2026-08-06 | 313 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# DIFFormer: Diffusion-based (Graph) Transformers
The official implementation for ICLR23 paper "DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion".
Related material: [[Conference Paper](https://arxiv.org/pdf/2301.09474.pdf) | [Extended Paper](https://arxiv.org/pdf/2409.09111)], [Blog [Chinese](https://zhuanlan.zhihu.com/p/622970740) | [English](https://medium.com/towards-data-science/how-to-build-graph-transformers-with-o-n-complexity-d507e103d30a)], [[Video](https://www.bilibili.com/video/BV1dP411C7Ti/?share_source=copy_web&vd_source=28f0a1823e05d5df3685cb9737bba371)]
DIFFormer is a general-purpose encoder that can be used to compute instance representations with their latent/observed interactions accommodated.
This work is built upon [NodeFormer](https://github.com/qitianwu/NodeFormer) (NeurIPS22) which is a scalable Transformer for large graphs with linear complexity.
## What's news
[2023.03.01] We release the early version of our codes for node classification.
[2023.03.09] We release codes for image/text classification and spatial-temporal prediction.
[2023.07.03] I gave a talk on LOG seminar about scalable graph Transformers. See the online video [here](https://www.bilibili.com/video/BV1dP411C7Ti/?share_source=copy_web&vd_source=28f0a1823e05d5df3685cb9737bba371).
[2024.08.15] We add codes for heterophily graphs (chameleon, squirrel and actor). See the training script in `./node classification/run.sh`.
[2024.08.15] We extend the implementation for accommodating a batch of graphs as input, which is the common case for graph-level predictive tasks. See codes for particle physics datasets in `./physical particle` and model implementation in `./physical particle/difformer-v2.py`.
[2025.06.20] The [extended paper](https://arxiv.org/pdf/2409.09111) is accepted to JMLR.
## Model Overview
DIFFormer is motivated by an energy-constrained diffusion process which encodes a batch of instances to their structured representations. At each step, the model will first estimate pair-wise influence (i.e., attention) among arbitrary instance pairs (regardless of whether they connected by an input graph) and then update instance embeddings by feature propagation. The feed-forward process can be treated as a diffusion process that minimizes the global energy.
<img width="700" alt="image" src="https://user-images.githubusercontent.com/22075007/232401434-e433a273-2083-4ac8-ad82-e9e15dd51d49.png">
In specific, the DIFFormer's architecture is depicted by the following figure where one DIFFormer layer comprises of global attention, GCN convolution and residual link. The global attention is our key design including two instantiations: DIFFormer-s and DIFFormer-a.
<img width="700" alt="image" src="https://files.mdnice.com/user/23982/0f71e990-acbc-4706-aca3-680628f8ac92.png">
We implement the model in `difformer.py` where the DIFFormer-s (resp. DIFFormer-a) corresponds to `kernel = 'simple' (resp. 'sigmoid')`. The differences of two model versions lie in the global attention computation where DIFFormer-s only requires $O(N)$ complexity and DIFFormer-a requires $O(N^2)$, illustrated by the figure below where the red color marks the computation bottleneck.
<img width="700" alt="image" src="https://files.mdnice.com/user/23982/3c433a8d-faf4-45f7-a4bd-c599e3288077.png">
## Where DIFFormer can be used?
We demonstrate the model on four different types of tasks: graph-based node classification, image and text classification, spatial-temporal prediction and particle property prediction.
Beyond these scenarios, DIFFormer can be used as a general-purpose encoder for various applications including but not limited to:
- 1): **Encoding a graph**: given node features $X$ and graph adjacency $A$, the model outputs node embeddings $Z$ or predictions $\hat Y$. Please refer to codes in `./node classification` or `./spatial-temporal` for how to use it.
```python
model = DIFFormer(in_channels, hidden_channels, Excerpt of 8,568 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:cbad86fce14e4d7c, topic:text-classification, readme:text classification
matched fp:cbad86fce14e4d7c, topic:pytorch
matched fp:cbad86fce14e4d7c, topic:transformer
matched fp:cbad86fce14e4d7c, topic:image-classification