Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Reproduction of OpenAI and DeepMind's "Deep Reinforcement Learning from Human Preferences"
| Date | Stars |
|---|---|
| 2026-07-31 | 337 |
| 2026-08-02 | 337 |
| 2026-08-03 | 337 |
| 2026-08-06 | 337 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Deep Reinforcement Learning from Human Preferences Reproduction of OpenAI and DeepMind's [Deep Reinforcement Learning from Human Preferences](https://blog.openai.com/deep-reinforcement-learning-from-human-preferences/), based on the paper at <https://arxiv.org/pdf/1706.03741.pdf>. ## Results The main milestones of this reproduction were: * Training an agent to move the dot to the middle in a [simple environment](https://github.com/mrahtz/gym-moving-dot) using synthetic preferences. * Training an agent to play Pong using synthetic preferences. * Training an agent to stay alongside other cars in Enduro using *human* preferences.    ## Usage ### Python setup This project uses Tensorflow 1, which needs Python 3.7 or below. To set up an isolated environment and install dependencies, install [Pipenv](https://github.com/pypa/pipenv), then just run: `$ pipenv install` However, note that TensorFlow must be installed manually. Either: `$ pipenv run pip install tensorflow==1.15` or `$ pipenv run pip install tensorflow-gpu==1.15` depending on whether you have a GPU. (If you run into problems, try installing TensorFlow 1.6.0, which was used for development.) If you want to run tests, also run: `$ pipenv install --dev` Finally, before running any of the scripts, enter the environment with: `$ pipenv shell` ### Running All training is done using [`run.py`](run.py). Basic usage is: `$ python3 run.py <mode> <environment>` Supported environments are [`MovingDotNoFrameskip-v0`](https://github.com/mrahtz/gym-moving-dot), `PongNoFrameskip-v4`, and `EnduroNoFrameskip-v4`. ### Training with original rewards To train using the original rewards from the environment rather than rewards based on preferences, use the `train_policy_with_original_rewards` mode. For example, to train Pong: `$ python3 run.py train_policy_with_original_rewards PongNoFrameskip-v4 --n_envs 16 --million_timesteps 10` ### Training end-to-end with preferences Use the `train_policy_with_preferences` mode. For example, to train `MovingDotNoFrameskip-v0` using *synthetic* preferences: `$ python3 run.py train_policy_with_preferences MovingDotNoFrameskip-v0 --synthetic_prefs --ent_coef 0.02 --million_timesteps 0.15` On a machine with a GPU, this takes about an hour. TensorBoard logs (created in a new directory in `runs/` automatically) should look something like:  To train Pong using *synthetic* preferences: `$ python3 run.py train_policy_with_preferences PongNoFrameskip-v4 --synthetic_prefs --dropout 0.5 --n_envs 16 --million_timesteps 20` On a 16-core machine without GPU, this takes about 13 hours. TensorBoard logs should look something like:  To train Enduro (a modified version with a time limit so the weather doesn't change, which the paper notes can confuse the reward predictor) using *human* preferences: `$ python3 run.py train_policy_with_preferences EnduroNoFrameskip-v4 --n_envs 16 --render_episodes` You'll see two windows: a larger one showing a pair of examples of agent behaviour, and another smaller window showing the last full episode that the agent played (so you can see how qualitative behaviour is changing). Enter 'L' in the terminal to indicate that you prefer the left example; 'R' to indicate you prefer the right example; 'E' to indicate you prefer them both equally; and just press enter if the two clips are incomparable. On an 8-core machine with GPU, it takes about 2.5 hours to reproduce the video above - about an hour to collect 500 preferences about behaviour from a random policy, then half an hour to pretrain the reward predictor using those 500 preferences, then an hour to train the policy (while still collecting preferences.) The bottleneck is mainly labelling speed, so if you're already saved human preferences in `runs/enduro`, you can re-use those preferences by training with: `$ python
Excerpt of 13,972 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:c4e80b9fe20a3ca1, llm:Repository description: 'Reproduction of OpenAI and DeepMind's "Deep Reinforcement Learning from Human Preferences"' (reinforcement learning from human preferences reproduction).