Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript
| Date | Stars |
|---|---|
| 2026-07-24 | 626 |
| 2026-07-25 | 626 |
| 2026-07-28 | 626 |
| 2026-07-30 | 626 |
| 2026-08-06 | 626 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# TokenMonster **UPDATE:** [Benchmark results from pretraining 16 language models on different tokenizers.](./benchmark/pretrain.md) TokenMonster is an ungreedy subword tokenizer and vocabulary generator, enabling language models to run faster, cheaper, smarter and generate longer streams of text. <img width="661" alt="tokenmonster" src="https://github.com/alasdairforsythe/tokenmonster/assets/77910352/1136330a-bf25-4a17-8edb-06b90fffb236"> Large and sub-optimal vocabularies lead to the waste of computational and memory resources in language models. By switching to TokenMonster, you can potentially achieve the same or better performance with a vocabulary that is [less than a quarter of the size](https://alasdair.com/tokenmonster/benchmark.html?a=tiktoken%20cl100k_base&b=englishcode-24000-clean-v1&c=englishcode-16000-unfiltered-v1). TokenMonster can train and generate an optimal vocabulary on a 1 GB dataset within 24 hours on a typical desktop. 442 [pretrained vocabularies](#pretrained-vocabularies) are provided, as well as tools to train your own vocabularies & implementations in Go, Python & Javascript for tokenization and detokenization using the pretrained or your own vocabularies. You can [test TokenMonster in your browser here](https://alasdair.com/tokenmonster/), tokenizing live in native Javascript. TokenMonster is a novel approach to tokenization with broad-ranging use potential, but its primary motivation is to improve the training, inference and context-length of large language models. By using a more optimal vocabulary and ungreedy tokenization algorithm, text can be represented with [37.5% fewer tokens at the same vocabulary size](https://alasdair.com/tokenmonster/benchmark.html?a=gpt2%20tokenmonster&b=tiktoken%20p50k_base&c=englishcode-50256-clean-v1) compared to other modern tokenizing methods, increasing the speed of inference, training and the length of text. And/or the vocabulary size can be [reduced by 75% or more](https://alasdair.com/tokenmonster/benchmark.html?a=tiktoken%20cl100k_base&b=tiktoken%20p50k_base&c=englishcode-24000-clean-v1), freeing resources that can be used to make the model smarter and faster. You can also import existing vocabularies from other tokenizers, allowing you to take advantage of TokenMonster's fast, ungreedy tokenization whilst still using the existing vocabulary your model was trained for. TokenMonster vocabularies for GPT2 Tokenizer and LLaMa Tokenizer are included. ## Features - Outperforms other tokenization algorithms in every area ([benchmark](./benchmark)) - Selects the optimal vocabulary for a given dataset - 5 [optimization modes](#optimization-modes) to choose from: `unfiltered`, `clean`, `balanced`, `consistent`, `strict` - Ungreedy: follows up to 6 parallel branches at a time - Fast: follows 6 branches faster than other algorithms can follow 1 ([benchmark](./benchmark)) - Utilizes [capcode](#capcode) marker tokens to encode uppercasing and forward delete - Successfully identifies words, subwords, common phrases and figures of speech by itself - Works with HTML tags, sequential spaces, tabs, etc. without wasting context - Can be trained on any language - Achieves up to 7 chr/token (depending on vocabulary size & optimization mode) - Vocabularies can be modified and resized after training - Full support for "special" and "single-byte" tokens - Import and export vocabularies to and from human-readable YAML format - 422 pretrained vocabularies ready for use ## Table of Contents * Usage [Go](./go/) | [Python](./python/) | [Javascript](./javascript/) | [Training](./training/) * [Benchmark](./benchmark) * [Pretrained Vocabularies](#pretrained-vocabularies) * [Optimization Modes](#optimization-modes) * [Vocabulary Selection Guidance](#vocabulary-selection-guidance) * [Capcode](#capcode) * [Normalization](#normalization) * [How does it work and how is it different from BPE?](#how-does-it-work-and-how-is-it-different-from-bpe) * [The Ungreedy Tokenization Algorithm](
Excerpt of 19,470 characters
Read on GitHub
203
Ryan Brooks
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:36054376db4dd07b, topic:tokenizer, desc:tokenizer, readme:tokenizer