Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[NeurIPS 2021] Galerkin Transformer: Neural Operator built on Attention for PDEs
| Date | Stars |
|---|---|
| 2026-07-24 | 266 |
| 2026-07-25 | 266 |
| 2026-07-28 | 266 |
| 2026-07-30 | 266 |
| 2026-08-06 | 266 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# [NeurIPS 2021] Galerkin Transformer: linear attention without softmax [](./LICENSE) [](https://www.python.org/downloads/release/python-370/) [](https://pytorch.org/) [](https://arxiv.org/abs/2105.14995) [](https://open.vscode.dev/scaomath/galerkin-transformer) # Summary - A non-numerical analyst oriented explanation on Toward Data Science about the [Galerkin Transformer](https://towardsdatascience.com/galerkin-transformer-a-one-shot-experiment-at-neurips-2021-96efcbaefd3e) - [The post on my blog](https://scaomath.github.io/blog/galerkin-transformer/), has much more details on the math of how to bridge the attention operator (a nonlinear operator)'s approximation capacity with a linear operator (Petrov-Galerkin projection). - The post on my mentor's WeChat blog CAM传习录 (in Chinese): [Galerkin Transformer: 初学者的进击](https://mp.weixin.qq.com/s?__biz=MzUxNzk0NjExOA==&mid=2247487695&idx=1&sn=be0c364d5d85ca83ee27f4425d0a38c2) - For how to train our models please refer to [the training instructions under the `/examples` folder](./examples/). - If just wanting to see what is it like for the models to perform on the unseen test set, please refer to [evaluation](#evaluation-notebooks). ## Introduction The new simple attention operator (for the encoder) is simply `Q(K^TV)` (Galerkin), or the quadratic complexity one `(QK^T)V` (Fourier). - No softmax, or the approximation thereof, at all. - Whichever two latent representations doing `matmul` get the layer normalization, similar to Gram-Schmidt process where we have to divide the basis's norm squared. `Q, K` get layer normalized in the Fourier-type attention (every position attends with every other), as for `K, V` in the Galerkin-type attention (every basis attends with every other basis). No layer normalization is applied afterward. - Some other components are tweaked according to our Hilbertian interpretation of attention. For the full operator learner, the feature extractor is a simple linear layer or an interpolation-based CNN, the decoder is the spectral convolution real parameter re-implementation from the best operator learner to-date Fourier Neural Operator (FNO) in [*Li et al 2020*](https://github.com/zongyi-li/fourier_neural_operator) if the target is smooth, or just a pointwise FFN if otherwise. The resulting network is extremely powerful in learning PDE-related operators (energy decay, inverse coefficient identification). ## Hilbertian framework to analyze a linear attention variant Even though everyone is Transformer'ing, the mathematics behind the attention mechanism is not well understood. We have also shown that the Galerkin-type attention (a linear attention without softmax) has an approximation capacity on par with a Petrov-Galerkin projection under a Hilbertian setup. We use a method commonly known as ''mixed method'' in the finite element analysis community that is used to solve fluid/electromagnetics problems. Unlike finite element methods, in an attention-based operator learner the approximation is not discretization-tied, in that: 1. The latent representation is interpreted "column-wise" (each column represents a basis), opposed to the conventional "row-wise"/ "position-wise"/"word-wise" interpretation of attention in NLP. 2. The dimensions of the approximation spaces are not tied to the geometry as in the traditional finite element analysis (or finite difference, spectral methods, radial basis, etc.); 3. The approximation spaces are being dynamically updated by the nonlinear universal approximator due to the presence of the positional encodings, which determines the topology of the approximation space. ## Interpretatio
Excerpt of 12,746 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:0f8e9b1640b9b6e4, topic:pytorch
matched fp:0f8e9b1640b9b6e4, topic:transformer