Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A state-of-the-art semi-supervised method for image recognition
| Date | Stars |
|---|---|
| 2026-07-24 | 1670 |
| 2026-07-25 | 1670 |
| 2026-07-28 | 1670 |
| 2026-07-30 | 1670 |
| 2026-08-06 | 1670 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Mean teachers are better role models [Paper](https://arxiv.org/abs/1703.01780) ---- [NIPS 2017 poster](nips_2017_poster.pdf) ---- [NIPS 2017 spotlight slides](nips_2017_slides.pdf) ---- [Blog post](https://thecuriousaicompany.com/mean-teacher/) By Antti Tarvainen, Harri Valpola ([The Curious AI Company](https://thecuriousaicompany.com/)) ## Approach Mean Teacher is a simple method for semi-supervised learning. It consists of the following steps: 1. Take a supervised architecture and make a copy of it. Let's call the original model the **student** and the new one the **teacher**. 2. At each training step, use the same minibatch as inputs to both the student and the teacher but add random augmentation or noise to the inputs separately. 3. Add an additional *consistency cost* between the student and teacher outputs (after softmax). 4. Let the optimizer update the student weights normally. 5. Let the teacher weights be an exponential moving average (EMA) of the student weights. That is, after each training step, update the teacher weights a little bit toward the student weights. Our contribution is the last step. Laine and Aila [\[paper\]](https://arxiv.org/abs/1610.02242) used shared parameters between the student and the teacher, or used a temporal ensemble of teacher predictions. In comparison, Mean Teacher is more accurate and applicable to large datasets.  Mean Teacher works well with modern architectures. Combining Mean Teacher with ResNets, we improved the state of the art in semi-supervised learning on the ImageNet and CIFAR-10 datasets. ImageNet using 10% of the labels | top-5 validation error ---------------------------------|------------------------ Variational Auto-Encoder [\[paper\]](https://arxiv.org/abs/1609.08976) | 35.42 ± 0.90 Mean Teacher ResNet-152 | **9.11 ± 0.12** All labels, state of the art [\[paper\]](https://arxiv.org/pdf/1709.01507.pdf) | 3.79 CIFAR-10 using 4000 labels | test error -----------------------------|----------- CT-GAN [\[paper\]](https://openreview.net/forum?id=SJx9GQb0-) | 9.98 ± 0.21 Mean Teacher ResNet-26 | **6.28 ± 0.15** All labels, state of the art [\[paper\]](https://arxiv.org/abs/1705.07485) | 2.86 ## Implementation There are two implementations, one for TensorFlow and one for PyTorch. The PyTorch version is probably easier to adapt to your needs, since it follows typical PyTorch idioms, and there's a natural place to add your model and dataset. Let me know if anything needs clarification. Regarding the results in the paper, the experiments using a traditional ConvNet architecture were run with the TensorFlow version. The experiments using residual networks were run with the PyTorch version. ## Tips for choosing hyperparameters and other tuning Mean Teacher introduces two new hyperparameters: EMA decay rate and consistency cost weight. The optimal value for each of these depends on the dataset, the model, and the composition of the minibatches. You will also need to choose how to interleave unlabeled samples and labeled samples in minibatches. Here are some rules of thumb to get you started: * If you are working on a new dataset, it may be easiest to start with only labeled data and do pure supervised training. Then when you are happy with the architecture and hyperparameters, add mean teacher. The same network should work well, although you may want to tune down regularization such as weight decay that you have used with small data. * Mean Teacher needs some noise in the model to work optimally. In practice, the best noise is probably random input augmentations. Use whatever relevant augmentations you can think of: the algorithm will train the model to be invariant to them. * It's useful to dedicate a portion of each minibatch for labeled examples. Then the supervised training signal is strong enough early on to train quickly and prevent getting stuck into uncertainty. In the PyTorch examples we have a qua
Excerpt of 5,347 characters
Read on GitHub
3
Christian S. Perone · United Kingdom
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:a7f5690e004d9b94, topic:deep-learning, topic:pytorch, topic:tensorflow