Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Implementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmind
| Date | Stars |
|---|---|
| 2026-07-31 | 406 |
| 2026-08-01 | 406 |
| 2026-08-02 | 406 |
| 2026-08-06 | 406 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<img src="./q-transformer.png" width="450px"></img>
## Q-transformer
Implementation of <a href="https://qtransformer.github.io/">Q-Transformer</a>, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmind
I will be keeping around the logic for Q-learning on single action just for final comparison with the proposed autoregressive Q-learning on multiple actions. Also to serve as education for myself and the public.
The autoregressive Q-learning formulation has been reproduced by [Kotb et al.](https://arxiv.org/abs/2407.18841)
[Q Learning is Not Yet Scalable](https://seohong.me/blog/q-learning-is-not-yet-scalable)
## Install
```bash
$ pip install q-transformer
```
## Usage
```python
import torch
from q_transformer import (
QRoboticTransformer,
QLearner,
Agent,
ReplayMemoryDataset
)
# the attention model
model = QRoboticTransformer(
vit = dict(
num_classes = 1000,
dim_conv_stem = 64,
dim = 64,
dim_head = 64,
depth = (2, 2, 5, 2),
window_size = 7,
mbconv_expansion_rate = 4,
mbconv_shrinkage_rate = 0.25,
dropout = 0.1
),
num_actions = 8,
action_bins = 256,
depth = 1,
heads = 8,
dim_head = 64,
cond_drop_prob = 0.2,
dueling = True
)
# you need to supply your own environment, by overriding BaseEnvironment
from q_transformer.mocks import MockEnvironment
env = MockEnvironment(
state_shape = (3, 6, 224, 224),
text_embed_shape = (768,)
)
# env.init() should return instructions and initial state: Tuple[str, Tensor[*state_shape]]
# env(actions) should return rewards, next state, and done flag: Tuple[Tensor[()], Tensor[*state_shape], Tensor[()]]
# agent is a class that allows the q-model to interact with the environment to generate a replay memory dataset for learning
agent = Agent(
model,
environment = env,
num_episodes = 1000,
max_num_steps_per_episode = 100,
)
agent()
# Q learning on the replay memory dataset on the model
q_learner = QLearner(
model,
dataset = ReplayMemoryDataset(),
num_train_steps = 10000,
learning_rate = 3e-4,
batch_size = 4,
grad_accum_every = 16,
)
q_learner()
# after much learning
# your robot should be better at selecting optimal actions
video = torch.randn(2, 3, 6, 224, 224)
instructions = [
'bring me that apple sitting on the table',
'please pass the butter'
]
actions = model.get_optimal_actions(video, instructions)
```
## Appreciation
- <a href="https://stability.ai/">StabilityAI</a>, <a href="https://a16z.com/supporting-the-open-source-ai-community/">A16Z Open Source AI Grant Program</a>, and <a href="https://huggingface.co/">🤗 Huggingface</a> for the generous sponsorships, as well as my other sponsors, for affording me the independence to open source current artificial intelligence research
## Todo
- [x] first work way towards single action support
- [x] offer batchnorm-less variant of maxvit, as done in SOTA weather model metnet3
- [x] add optional deep dueling architecture
- [x] add n-step Q learning
- [x] build the conservative regularization
- [x] build out main proposal in paper (autoregressive discrete actions until last action, reward given only on last)
- [x] improvise decoder head variant, instead of concatenating previous actions at the frames + learned tokens stage. in other words, use classic encoder - decoder
- [x] allow for cross attention to fine frame / learned tokens
- [x] redo maxvit with axial rotary embeddings + sigmoid gating for attending to nothing. enable flash attention for maxvit with this change
- [x] build out a simple dataset creator class, taking in the environment and model and returning a folder that can be accepted by a `ReplayDataset`
- [x] finish basic environment loop
- [x] store memories to memmapped files in designated folder
- [x] `ReplayDataset` that takes in folder
- [x] 1 time step option
Excerpt of 7,728 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:6cc5ee0b75c11913, llm:Repository topics and description: 'Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions' (topics include q-learning, reinforcement, transformers, robotics, attention-mechanisms, deep-learning).
matched fp:6cc5ee0b75c11913, llm:Repository topics and description: 'Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions' (topics include q-learning, reinforcement, transformers, robotics, attention-mechanisms, deep-learning).
matched fp:6cc5ee0b75c11913, llm:Repository topics and description: 'Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions' (topics include q-learning, reinforcement, transformers, robotics, attention-mechanisms, deep-learning).