Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
An alternative to convolution in neural networks
| Date | Stars |
|---|---|
| 2026-07-31 | 264 |
| 2026-08-05 | 264 |
| 2026-08-06 | 264 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Sharpened Cosine Similarity An alternative to convolution for neural networks ## Implementations [PyTorch](https://github.com/brohrer/sharpened-cosine-similarity/tree/main/pytorch) [Keras](https://github.com/brohrer/sharpened-cosine-similarity/tree/main/keras) [Jax](https://github.com/brohrer/sharpened-cosine-similarity/tree/main/jax) ## Description Sharpened cosine similarity is a strided operation, like convolution, that extracts features from an image. It is related to convolution, but with important defferences. Convolution is a strided dot product between a signal, *s*, and a kernel *k*.  A cousin of convolution is cosine similarity, where the signal patch and kernel are both normalized to have a magnitude of 1 before the dot product is taken. It is so named because in two dimensions, it gives the cosine of the angle between the signal and the kernel vectors.  The cosine is known for being broad, that is, two quite different vectors can have a moderately high cosine similarity. It can be sharpened by raising the magnitude of the result to a power, *p*, while maintaining the sign.  This measure can become numerically unstable if ever the magnitude of the signal or kernel gets too close to zero. Adding a small value, *q*, to the signal magnitude. In practice, the kernel magnitude doesn't get too small and doesn't need this term.  ## Background The idea behind sharpened cosine similarity first surfaced as [a Twitter thread](https://twitter.com/_brohrer_/status/1232063619657093120) in 2020. There's some more development in this [blog post](https://www.rpisoni.dev/posts/cossim-convolution-part2/). ## Tips and Tricks These are some things that have been reported to work so far. * The big benefit of SCS appears to be parameter efficiency and architecture simplicity. It doesn't look like it's going to beat any accuracy records, and it doesn't always run very fast, but it's killing in this [parameter efficiency leaderboard](https://github.com/brohrer/parameter_efficiency_leaderboard). * Skip the nonlinear activation layers, like ReLU and sigmoid, after SCS layers. * Skip the dropout layers after SCS layers. * Skip the normalization layers, like batch normalization or layer normalization, after SCS layers. * Use MaxAbsPool instead of MaxPool. It selects the element with the highest magnitude of activity, even if it's negative. * Raising activities to the power p generally doesn't parallelize well on GPUs and TPUs. It will slow your code down a LOT compared to straight convolutions. Disabling the p parameters results in a huge speedup on GPUs, but this takes the "sharpened" out of SCS. Regular old cosine similarity is cool, but it is its own thing with its own limitations. ## Examples In the age of gargantuan language models, it's uncommon to talk about how *few* parameters a model uses, but it matters when you hope to deploy on compute- or power-limited devices. Sharpened cosine similarity is exceptionally parameter efficient. The repository <a href="https://github.com/brohrer/scs_torch_gallery">scs_torch_gallery</a> has a handful of working examples. [`cifar10_80_25214.py`](https://github.com/brohrer/scs-gallery/blob/main/cifar10_80_25214.py) is an image classification model that gets 80% accuracy on CIFAR 10, using only 25.2k parameters. According to the [CIFAR-10 Papers With Code](https://paperswithcode.com/sota/image-classification-on-cifar-10?dimension=PARAMS) this is somewhere around one-tenth of the parameters in previous models in this accuracy range. ## Reverse Chronology | Date | Milestone | | ------------- | ------------- | | 2022-12-06 | [Paper](https://openreview.net/forum?id=8z_L0LTx9Q) by Skyler Wu, Fred Lu, Edward Raff, James Holt in NeurIPS 2022 [ICBINB Workshop](https://
Excerpt of 8,153 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:db38b4f72b307665, topic:image-classification, readme:image classification