Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[ICLR2025 Spotlight🔥] Official Implementation of TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters
| Date | Stars |
|---|---|
| 2026-07-24 | 593 |
| 2026-07-25 | 593 |
| 2026-07-28 | 593 |
| 2026-07-30 | 593 |
| 2026-08-06 | 593 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# TokenFormer: a fully attention-based neural network with tokenized model parameters. Maximizing the flexibility of Transformer by Tokenizing Anything. <h5 align="center"> [](https://arxiv.org/abs/2410.23168) [](https://haiyang-w.github.io/tokenformer.github.io/) [](https://huggingface.co/Haiyang-W) [](https://hits.seeyoufarm.com) </h5> This repo is the official implementation of our paper: [TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters](https://arxiv.org/abs/2410.23168) as well as the follow-ups. Our TokenFormer is a natively scalable architecture that leverages the attention mechanism not only for computations among input tokens but also for interactions between tokens and model parameters, thereby enhancing architectural flexibility. We have made every effort to ensure that the codebase is clean, concise, easily readable, state-of-the-art, and relies only on minimal dependencies. > TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters > > [Haiyang Wang*](https://haiyang-w.github.io/), [Yue Fan*](https://yue-fan.github.io/), [Muhammad Ferjad Naeem](https://ferjad.github.io/), [Yongqin Xian](https://xianyongqin.github.io/), [Jan Eric Lenssen](https://janericlenssen.github.io/), [Liwei Wang](http://www.liweiwang-pku.com/), [Federico Tombari](https://federicotombari.github.io/), [Bernt Schiele](https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/people/bernt-schiele) > - Primary contact: Haiyang Wang ~~([email protected])~~([email protected]), Bernt Schiele ([email protected]) <div align="center"> <img src="assets/Figure1.png" width="800"/> </div> ## 📣 News - [25-02-11] 🔥 TokenFormer is accepted as **[spotlight](https://openreview.net/forum?id=oQ4igHyh3N)** presentation. - [25-01-22] 🔥 TokenFormer is accepted by **[ICLR2025](https://iclr.cc/)**. - [25-01-12] Jax code on TPU (GCP-Cloud) is released, please see [here](https://github.com/Haiyang-W/TokenFormer/tree/main/tokenformer-tpu). - [24-11-08] 🚀 Training code with pytorch is released. - [24-11-02] Please feel free to email me if I've missed any relevant papers. I will do my best to include all related papers in future versions. - [24-10-31] 🚀 Inference code with pytorch is released. - [24-10-31] 👀 TokenFormer is released on [arXiv](https://arxiv.org/abs/2410.23168). ## 🔥 Some Thoughts - We aim to offer a new perspective to models, applicable to any computation graph in the future. In theory, by using **data tokens**, **parameter tokens**, and **memory tokens**, and **through dot-product interactions**, it's possible to flexibly construct any network. There are many design possibilities here. For example, introducing memory tokens can build RNN-like networks similar to [Mamba](https://arxiv.org/abs/2312.00752). Merging parameter tokens with memory tokens creates something akin to a [TTT network](https://arxiv.org/abs/2407.04620). Parameter tokens can also attend to input data in reverse, making the network parameters dynamically data-dependent, updating layer by layer. ## Overview - [💫 What we want to do](https://github.com/Haiyang-W/TokenFormer#what-we-want-to-do) - [🚀 Main Results](https://github.com/Haiyang-W/TokenFormer#main-results) - [📘 Model Zoo](https://github.com/Haiyang-W/TokenFormer#model-zoo) - [🛠️ Quick Start](https://github.com/Haiyang-W/TokenFormer#quick-start) - [👀 Todo](https://github.com/Haiyang-W/TokenFormer#todo) - [📘 Citation](https://github.com/Haiyang-W/TokenFormer#citation)
Excerpt of 17,851 characters
Read on GitHub
23
Fan Yue · MPI-INF
2
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:dee6ea200a7cf1d1, topic:llm, topic:foundation-models, topic:transformer