Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Beyond Language Models: Byte Models are Digital World Simulators
| Date | Stars |
|---|---|
| 2026-07-31 | 334 |
| 2026-08-06 | 333 |
Today
-1 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Beyond Language Models: Byte Models are Digital World Simulators This repository contains the code for the bGPT model as described in the paper [Beyond Language Models: Byte Models are Digital World Simulators](https://arxiv.org/abs/2402.19155). bGPT supports generative modelling via next byte prediction on any type of data and can perform any task executable on a computer, showcasing the capability to simulate all activities within the digital world, with its potential only limited by computational resources and our imagination. You can check out the [demo page](https://byte-gpt.github.io/), which includes examples generated by the bGPT model. ## Model Description Traditional deep learning often overlooks bytes, the basic units of the digital world, where all forms of information and operations are encoded and manipulated in binary format. Inspired by the success of next token prediction in natural language processing, we introduce bGPT, a model with next byte prediction to simulate the digital world. bGPT matches specialized models in performance across various modalities, including text, audio, and images, and offers new possibilities for predicting, simulating, and diagnosing algorithm or hardware behaviour. It has almost flawlessly replicated the process of converting symbolic music data, achieving a low error rate of 0.0011 bits per byte in converting ABC notation to MIDI format. In addition, bGPT demonstrates exceptional capabilities in simulating CPU behaviour, with an accuracy exceeding 99.99% in executing various operations. Leveraging next byte prediction, models like bGPT can directly learn from vast binary data, effectively simulating the intricate patterns of the digital world. We provide five weights of bGPT on [Hugging Face](https://huggingface.co/sander-wood/bgpt/tree/main) corresponding to each dataset used for pre-training: 1. **_weights-conversion.pth_**: bGPT pre-trained on IrishMAN for data conversion (between `.abc` and `.mid`). 2. **_weights-cpu.pth_**: bGPT pre-trained on CPU states for CPU state modelling (`.bin`). 3. **_weights-text.pth_**: bGPT pre-trained on Wikipedia for text generation/classification (`.txt`). 4. **_weights-image.pth_**: bGPT pre-trained on ImageNet for image generation/classification (`.bmp`). 5. **_weights-audio.pth_**: bGPT pre-trained on Librispeech for audio generation/classification (`.wav`). The core components of bGPT include a 12-layer patch-level decoder, a 3-layer byte-level decoder, with a hidden size of 768, totaling 110 million parameters. ## Installation To set up the bGPT environment and install the necessary dependencies, follow these steps: 1. **Create and Activate Conda Environment** ```bash conda create --name bgpt python=3.7.9 conda activate bgpt ``` 2. **Install Dependencies** ```bash pip install -r requirements.txt ``` 3. **Install Pytorch** ```bash pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116 ``` 4. **Download Pre-trained bGPT Weights (Optional)** For those interested in starting with pre-trained models, bGPT weights are available on [Hugging Face](https://huggingface.co/sander-wood/bgpt/tree/main). This step is optional but recommended for users looking to leverage the model's capabilities without training from scratch. ## Usage - `config.py`: Configuration settings for training and inference. - `cpu-simulation.py`: Simulate CPU states and operations. - `inference.py`: Perform inference tasks (e.g., generation and conversion) using pre-trained models. - `train-cls.py`: Training script for classification models. - `train-gen.py`: Training script for generative models. - `utils.py`: Utility functions supporting model operations and data processing. ### Configuration The `config.py` file contains critical settings for training and inference, allowing for flexibility across different tasks and modalities. Here'
Excerpt of 17,906 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:310d812a100dce69, llm:Repository description: 'Beyond Language Models: Byte Models are Digital World Simulators' (no topics). Python project focused on models beyond language models—likely foundation models / experimental modeling of digital world; name 'bgpt' suggests model.
matched fp:310d812a100dce69, llm:Repository description: 'Beyond Language Models: Byte Models are Digital World Simulators' (no topics). Python project focused on models beyond language models—likely foundation models / experimental modeling of digital world; name 'bgpt' suggests model.