Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
| Date | Stars |
|---|---|
| 2026-07-24 | 825 |
| 2026-07-25 | 826 |
| 2026-07-28 | 826 |
| 2026-07-30 | 826 |
| 2026-08-09 | 830 |
| 2026-08-17 | 833 |
| 2026-08-19 | 834 |
| 2026-08-20 | 835 |
| 2026-08-21 | 836 |
| 2026-08-25 | 838 |
| 2026-08-28 | 839 |
| 2026-08-30 | 838 |
| 2026-08-31 | 839 |
| 2026-09-01 | 838 |
| 2026-09-04 | 839 |
| 2026-09-05 | 840 |
| 2026-09-06 | 841 |
| 2026-09-09 | 841 |
| 2026-09-11 | 840 |
| 2026-09-12 | 841 |
| 2026-09-13 | 842 |
| 2026-09-16 | 843 |
| 2026-09-18 | 845 |
| 2026-09-19 | 846 |
| 2026-09-20 | 846 |
Today
— stars today
This week
+4 stars this week
This month
+10 stars this month
Momentum
4.0
growth rate 0.47%/day
# gpt-tokenizer
[](https://www.npmjs.com/package/gpt-tokenizer)
[](https://www.npmjs.com/package/gpt-tokenizer)
[](https://opensource.org/licenses/MIT)
[](https://github.com/niieani/gpt-tokenizer/actions)
`gpt-tokenizer` is a TypeScript Byte Pair Encoder/Decoder for every OpenAI model family, including GPT-5, GPT-4.1, GPT-4o, o-series reasoning models, gpt-oss, realtime and audio models, and legacy GPT-2/3/4 models.
It's the [_fastest, smallest and lowest footprint_](#benchmarks) GPT tokenizer available for all JavaScript environments and is written in TypeScript.
> Try it out in the **[playground](https://gpt-tokenizer.dev/)**!
This library has been trusted by:
- Microsoft ([Teams](https://github.com/microsoft/teams-ai), [GenAIScript](https://github.com/microsoft/genaiscript/))
- Elastic ([Kibana](https://github.com/elastic/kibana))
- [Effect TS](https://effect.website/)
- [CodeRabbit](https://www.coderabbit.ai/)
- [Rivet](https://github.com/Ironclad/rivet) by Ironclad
Please consider [🩷 sponsoring](https://github.com/sponsors/niieani) the project if you find it useful.
#### Features
It is the most feature-complete, open-source GPT tokenizer on NPM. This package is a port of OpenAI's [tiktoken](https://github.com/openai/tiktoken), with some additional, unique features sprinkled on top:
- Support for easily tokenizing chats thanks to the `encodeChat` function
- Support for all current OpenAI models, including GPT-5, GPT-4.1, GPT-4o, o1/o3/o4, gpt-oss, realtime, audio, transcription, image, and video models
- All OpenAI encodings: `r50k_base`, `p50k_base`, `p50k_edit`, `cl100k_base`, `o200k_base`, and `o200k_harmony`
- Can be loaded and work synchronously! (i.e. in non async/await contexts)
- Generator function versions of both the decoder and encoder functions
- Provides the ability to decode an asynchronous stream of data (using `decodeAsyncGenerator` and `decodeGenerator` with any iterable input)
- No global cache (no accidental memory leaks, as with the original GPT-3-Encoder implementation)
- Includes a highly performant `isWithinTokenLimit` function to assess token limit without encoding the entire text/chat
- Built-in cost estimation with the `estimateCost` function for calculating API usage costs
- Full OpenAI model catalog with context limits, capabilities, modalities, and comprehensive pricing information (see [`src/models.ts`](./src/models.ts) and [`src/models.gen.ts`](./src/models.gen.ts))
- Improves overall performance by eliminating transitive arrays
- Type-safe (written in TypeScript)
- Works in the browser out-of-the-box
## Installation
### As NPM package
```bash
npm install gpt-tokenizer
```
### Directly in a browser
```html
<script type="module">
import { encode, decode } from 'https://unpkg.com/gpt-tokenizer/esm/main.js'
</script>
```
Pin a package version in production. Browser delivery uses the same ES modules as
the NPM package; global UMD bundles are no longer published.
## Playground
The playground is published under a memorable URL: https://gpt-tokenizer.dev/
[](https://gpt-tokenizer.dev/)
## Usage
The library provides various functions to transform text into (and from) a sequence of integers (tokens) that can be fed into an LLM model. The transformation is done using a Byte Pair Encoding (BPE) algorithm used by OpenAI.
```typescript
import {
encode,
encodeChat,
decode,
isWithinTokenLimit,
encodeGenerator,
decodeGenerator,
decodeAsyncGenerator,
ALL_SPECIAL_TOKENS,
} from 'gpt-tokenizer'
// note: depending on the model, import from the respective Excerpt of 19,604 characters
Read on GitHub
Bazyli Brzóska · @zendesk · United States
86
26
8
7
4
3
1
1
1
1
Andrew Healey
1
David Schnurr · OpenAI · United States
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:b9fd1f4549a0430b, topic:tokenizer, name:tokenizer, desc:tokenizer