Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
| Date | Stars |
|---|---|
| 2026-07-24 | 825 |
| 2026-07-25 | 826 |
| 2026-07-28 | 826 |
| 2026-07-30 | 826 |
| 2026-08-06 | 826 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# gpt-tokenizer
[](https://www.npmjs.com/package/gpt-tokenizer)
[](https://www.npmjs.com/package/gpt-tokenizer)
[](https://opensource.org/licenses/MIT)
[](https://github.com/niieani/gpt-tokenizer/actions)
`gpt-tokenizer` is a Token Byte Pair Encoder/Decoder supporting all OpenAI's models (including GPT-5, GPT-4o, o1, o3, o4, GPT-4.1 and older models like GPT-3.5, GPT-4).
It's the [_fastest, smallest and lowest footprint_](#benchmarks) GPT tokenizer available for all JavaScript environments and is written in TypeScript.
> Try it out in the **[playground](https://gpt-tokenizer.dev/)**!
This library has been trusted by:
- Microsoft ([Teams](https://github.com/microsoft/teams-ai), [GenAIScript](https://github.com/microsoft/genaiscript/))
- Elastic ([Kibana](https://github.com/elastic/kibana))
- [Effect TS](https://effect.website/)
- [CodeRabbit](https://www.coderabbit.ai/)
- [Rivet](https://github.com/Ironclad/rivet) by Ironclad
Please consider [🩷 sponsoring](https://github.com/sponsors/niieani) the project if you find it useful.
#### Features
It is the most feature-complete, open-source GPT tokenizer on NPM. This package is a port of OpenAI's [tiktoken](https://github.com/openai/tiktoken), with some additional, unique features sprinkled on top:
- Support for easily tokenizing chats thanks to the `encodeChat` function
- Support for all current OpenAI models (available encodings: `r50k_base`, `p50k_base`, `p50k_edit`, `cl100k_base`, `o200k_base`, and `o200k_harmony`)
- Can be loaded and work synchronously! (i.e. in non async/await contexts)
- Generator function versions of both the decoder and encoder functions
- Provides the ability to decode an asynchronous stream of data (using `decodeAsyncGenerator` and `decodeGenerator` with any iterable input)
- No global cache (no accidental memory leaks, as with the original GPT-3-Encoder implementation)
- Includes a highly performant `isWithinTokenLimit` function to assess token limit without encoding the entire text/chat
- Built-in cost estimation with the `estimateCost` function for calculating API usage costs
- Full library of OpenAI models with comprehensive pricing information (see [`src/models.ts`](./src/models.ts) and [`src/models.gen.ts`](./src/models.gen.ts))
- Improves overall performance by eliminating transitive arrays
- Type-safe (written in TypeScript)
- Works in the browser out-of-the-box
## Installation
### As NPM package
```bash
npm install gpt-tokenizer
```
### As a UMD module
```html
<script src="https://unpkg.com/gpt-tokenizer"></script>
<script>
// the package is now available as a global:
const { encode, decode } = GPTTokenizer_cl100k_base
</script>
```
If you wish to use a custom encoding, fetch the relevant script.
- https://unpkg.com/gpt-tokenizer/dist/o200k_base.js (for all modern models, such as `gpt-5`, `gpt-4o`, `gpt-4.1`, `o1` and others)
- https://unpkg.com/gpt-tokenizer/dist/o200k_harmony.js (for open-weight Harmony models such as `gpt-oss-20b` and `gpt-oss-120b`)
- https://unpkg.com/gpt-tokenizer/dist/cl100k_base.js (for `gpt-4` and `gpt-3.5`)
- https://unpkg.com/gpt-tokenizer/dist/p50k_base.js
- https://unpkg.com/gpt-tokenizer/dist/p50k_edit.js
- https://unpkg.com/gpt-tokenizer/dist/r50k_base.js
The global name is a concatenation: `GPTTokenizer_${encoding}`.
Refer to [supported models and their encodings](#Supported-models-and-their-encodings) section for more information.
## Playground
The playground is published under a memorable URL: https://gpt-tokenizer.dev/
[](https://gpt-tokenizer.dev/)
## Usage
The librExcerpt of 19,676 characters
Read on GitHub
Bazyli Brzóska · @zendesk · United States
86
26
8
7
4
3
1
1
1
1
Andrew Healey
1
David Schnurr · OpenAI · United States
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:b9fd1f4549a0430b, topic:tokenizer, name:tokenizer, desc:tokenizer