Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Effortless plugin and play Optimizer to cut model training costs by 50%. New optimizer that is 2x faster than Adam on LLMs.
| Date | Stars |
|---|---|
| 2026-07-31 | 382 |
| 2026-08-06 | 382 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
[](https://discord.gg/qUtxnK2NMf)
# Sophia Optimizer
[PAPER LINK: Sophia: A Scalable Stochastic Second-order Optimizer for
Language Model Pre-training](https://arxiv.org/pdf/2305.14342.pdf)
[Author Implementation](https://github.com/Liuhong99/Sophia)
Cut Model Training Cost by 50%? with this all-new simple plug in and play Optimizer: Sophia
# Usage
Download with pip ```pip install Sophia-Optimizer```
```python
import torch
from torch import nn
from Sophia import SophiaG
class MyModel(nn.Module):
def __init__(self):
super(MyModel, self).__init__()
self.fc1 = nn.Linear(784, 128)
self.fc2 = nn.Linear(128, 10)
def forward(self, x):
x = torch.relu(self.fc1(x))
x = self.fc2(x)
return x
#init model loss function and input data
model = MyModel()
loss_function = nn.CrossEntropy()
input_data = ... #input data
#init the optimizer
optimizer = SophiaG(model.parameters(), lr=2e-4, betas=(0.965, 0.99), rho = 0.01, weight_decay=1e-1)
#training loop
for epoch in range(epochs):
for batch in data_loader:
optimizer.zero_grad()
output = model(batch)
loss = loss_function(output, target)
loss.backward()
optimizer.step()
```
## Training:
To run training use git clone method
navigate to experiments folder
```cd Sophia```
```cd experiments```
then run file
```python3 training.py````
and if not then do the following:
```python
from Sophia import DecoupledSophia, trainer
#train model
trainer.train()
#eval the model
eval_results = trainer.evaluate()
print(f"Perplexity: {torch.exp(torch.tensor(eval_results['eval_loss']))}")
```
Now with training file ready in experiments folder! 🔥🔥🔥
Sophia is an second order clipped stochastic optimization algorithm that uses an inexpensive stochastic estimate of the diagonal of the Hessian as an pre-conditioner and a clipping mechanism to control the worst case update size. It achieves better performance than adam in terms of validation pre-traing loss, total compute, and wall-clock time. By cutting model training cost in half, Sophia can help save millions if not billions of dollars in computational resources.
## Benefits
Sophia achievs the same validation pre training loss with 50% fewer number of steps than Adam
50% less total compute and 50% less wall-clock time
Seamless integration into existing training pipelines -- plug in and play!
No special requirments on model architecture or computing infrastructure
Supports both Hutchinson and Gauss-Newton-Bartlett Hessian Estimators
# Algorithmic pseudocode:
```
Initialize parameters: θ1, learning rate {ηt}, hyperparameters λ, β1, β2, ϵ, and estimator choice Estimator ∈ {Hutchinson, Gauss-Newton-Bartlett}
Set m0 = 0, v0 = 0, h1−k = 0
For t = 1 to T do
Compute minibatch loss Lt(θt)
Compute gt = ∇Lt(θt)
mt = β1mt−1 + (1 − β1)gt
If t mod k = 1 then
Compute hˆt = Estimator(θt)
ht = β2ht−k + (1 − β2)hˆt
Else
ht = ht−1
θt = θt − ηtλθt (weight decay)
θt+1 = θt − ηt · clip(mt/ max{ht, ϵ}, ρ)
```
# Pytorch Implementation
```python
import torch
class Sophia(torch.optim.Optimizer):
def __init__(self, model, input_data, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0, k=10, estimator="Hutchinson", rho=1):
self.model = model
self.input_data = input_data
defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay, k=k, estimator=estimator, rho=rho)
super(Sophia, self).__init__(params, defaults)
def step(self, closure=None):
loss = None
if closure is not None:
loss = closure()
for group in self.param_groups:
for p in group["params"]:
if p.grad is None:
continue
grad = p.grad.data
if grad.is_sparse:
raise RuntimeError("Sophia does not support spaExcerpt of 11,974 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:e592ca90ed0f096b, topic:deep-learning, topic:neural-network
matched fp:e592ca90ed0f096b, topic:chatgpt