Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Structured state space sequence models
| Date | Stars |
|---|---|
| 2026-07-24 | 2911 |
| 2026-07-25 | 2911 |
| 2026-07-28 | 2911 |
| 2026-07-30 | 2911 |
| 2026-08-06 | 2911 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Structured State Spaces for Sequence Modeling This repository provides the official implementations and experiments for models related to [S4](https://arxiv.org/abs/2111.00396), including [HiPPO](https://arxiv.org/abs/2008.07669), [LSSL](https://arxiv.org/abs/2110.13985), [SaShiMi](https://arxiv.org/abs/2202.09729), [DSS](https://arxiv.org/abs/2203.14343), [HTTYH](https://arxiv.org/abs/2206.12037), [S4D](https://arxiv.org/abs/2206.11893), and [S4ND](https://arxiv.org/abs/2210.06583). Project-specific information for each of these models, including overview of the source code and specific experiment reproductions, can be found under [models/](models/). ## Table of Contents Setting up the environment and porting S4 to external codebases: - [Setup](#setup) - [Getting Started with S4](#getting-started-with-s4) Using this repository for training models: - [Training](#training) - [Generation](#generation) - [Repository Structure](#overall-repository-structure) - [Citation](#citation) ### Changelog See [CHANGELOG.md](CHANGELOG.md) ### Roadmap - More documentation for training from scratch using this repository - Compilation of S4 resources and implementations - pip package ## Setup ### Requirements This repository requires Python 3.9+ and Pytorch 1.10+. It has been tested up to Pytorch 1.13.1. Other packages are listed in [requirements.txt](./requirements.txt). Some care may be needed to make some of the library versions compatible, particularly torch/torchvision/torchaudio/torchtext. Example installation: ``` conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia pip install -r requirements.txt ``` ### Structured Kernels A core operation of S4 are the Cauchy and Vandermonde kernels described in the [paper](https://arxiv.org/abs/2111.00396). These are very simple matrix multiplications; a naive implementation of these operation can be found in the [standalone](models/s4/s4.py) in the function `cauchy_naive` and `log_vandermonde_naive`. However, as the paper describes, this has suboptimal memory usage that currently requires a custom kernel to overcome in PyTorch. Two more efficient methods are supported. The code will automatically detect if either of these is installed and call the appropriate kernel. #### Custom CUDA Kernel This version is faster but requires manual compilation for each machine environment. Run `python setup.py install` from the directory `extensions/kernels/`. #### Pykeops This version is provided by the [pykeops library](https://www.kernel-operations.io/keops/python/installation.html). Installation usually works out of the box with `pip install pykeops cmake` which are also listed in the requirements file. ## Getting Started with S4 ### S4 Module Self-contained files for the S4 layer and variants can be found in [models/s4/](./models/s4/), which includes instructions for calling the module. See [notebooks/](notebooks/) for visualizations explaining some concepts behind HiPPO and S4. ### Example Train Script (External Usage) [example.py](example.py) is a self-contained training script for MNIST and CIFAR that imports the standalone S4 file. The default settings `python example.py` reaches 88% accuracy on sequential CIFAR with a very simple S4D model of 200k parameters. This script can be used as an example for using S4 variants in external repositories. ### Training with this Repository (Internal Usage) This repository aims to provide a very flexible framework for training sequence models. Many models and datasets are supported. The basic entrypoint is `python -m train`, or equivalently ``` python -m train pipeline=mnist model=s4 ``` which trains an S4 model on the Permuted MNIST dataset. This should get to around 90% after 1 epoch which takes 1-3 minutes depending on GPU. More examples of using this repository are documented throughout. See [Training](#training) for an overview. ### Optimizer Hyperparameters One important feat
Excerpt of 12,512 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:e5c634af48137417, topic:pytorch
