Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Tiny JavaScript tokenizer.
| Date | Stars |
|---|---|
| 2026-07-24 | 556 |
| 2026-07-25 | 556 |
| 2026-07-28 | 556 |
| 2026-07-30 | 556 |
| 2026-08-06 | 556 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# js-tokens
The tiny, regex powered, lenient, _almost_ spec-compliant JavaScript tokenizer that never fails.
```js
import jsTokens from "js-tokens";
const jsString = 'JSON.stringify({k:3.14**2}, null /*replacer*/, "\\t")';
Array.from(jsTokens(jsString), (token) => token.value).join("|");
// JSON|.|stringify|(|{|k|:|3.14|**|2|}|,| |null| |/*replacer*/|,| |"\t"|)
```
## Installation
`npm install js-tokens`
```js
import jsTokens from "js-tokens";
// or:
const jsTokens = require("js-tokens"); // Requires Node.js 20.19.0 or later.
```
## Usage
```js
jsTokens(string, options?)
```
| Option | Type | Default | Description |
| :----- | :-------- | :------ | :------------------ |
| jsx | `boolean` | `false` | Enable JSX support. |
This package exports a generator function, `jsTokens`, that turns a string of JavaScript code into token objects.
For the empty string, the function yields nothing (which can be turned into an empty list). For any other input, the function always yields _something,_ even for invalid JavaScript, and never throws. Concatenating the token values reproduces the input.
The package is very close to being fully spec compliant (it passes all [but 3](#known-errors) of [test262-parser-tests]), but has taken a couple of shortcuts. See the following sections for limitations of some tokens.
```js
// Loop over tokens:
for (const token of jsTokens("hello, !world")) {
console.log(token);
}
// Get all tokens as an array:
const tokens = Array.from(jsTokens("hello, !world"));
```
## Tokens
_Spec: [ECMAScript Language: Lexical Grammar] + [Additional Syntax]_
```ts
export default function jsTokens(input: string): Iterable<Token>;
type Token =
| { type: "StringLiteral"; value: string; closed: boolean }
| { type: "NoSubstitutionTemplate"; value: string; closed: boolean }
| { type: "TemplateHead"; value: string }
| { type: "TemplateMiddle"; value: string }
| { type: "TemplateTail"; value: string; closed: boolean }
| { type: "RegularExpressionLiteral"; value: string; closed: boolean }
| { type: "MultiLineComment"; value: string; closed: boolean }
| { type: "SingleLineComment"; value: string }
| { type: "HashbangComment"; value: string }
| { type: "IdentifierName"; value: string }
| { type: "PrivateIdentifier"; value: string }
| { type: "NumericLiteral"; value: string }
| { type: "Punctuator"; value: string }
| { type: "WhiteSpace"; value: string }
| { type: "LineTerminatorSequence"; value: string }
| { type: "Invalid"; value: string };
```
### StringLiteral
_Spec: [StringLiteral]_
If the ending `"` or `'` is missing, the token has `closed: false`. JavaScript strings cannot contain (unescaped) newlines, so unclosed strings simply end at the end of the line.
Escape sequences are supported, but may be invalid. For example, `"\u"` is matched as a StringLiteral even though it contains an invalid escape.
Examples:
<!-- prettier-ignore -->
```js
"string"
'string'
""
''
"\""
'\''
"valid: \u00a0, invalid: \u"
'valid: \u00a0, invalid: \u'
"multi-\
line"
'multi-\
line'
" unclosed
' unclosed
```
### NoSubstitutionTemplate / TemplateHead / TemplateMiddle / TemplateTail
_Spec: [NoSubstitutionTemplate] / [TemplateHead] / [TemplateMiddle] / [TemplateTail]_
A template without interpolations is matched as is. For, example:
- `` `abc` ``: NoSubstitutionTemplate
- `` `abc ``: NoSubstitutionTemplate with `closed: false`
A template _with_ interpolations is matched as many tokens. For example, `` `head${1}middle${2}tail` `` is matched as follows (apart from the two NumericLiterals):
- `` `head${ ``: TemplateHead
- `}middle${`: TemplateMiddle
- `` }tail` ``: TemplateTail
TemplateMiddle is optional, and TemplateTail can be unclosed. For example, `` `head${1}tail `` (note the missing ending `` ` ``):
- `` `head${ ``: TemplateHead
- `}tail`: TemplateTail with `closed: false`
Templates can contain unescaped newlines, so unclosed templates go on to the end of input.
Just like forExcerpt of 18,492 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:986d41f422b50405, topic:tokenizer, desc:tokenizer, readme:tokenizer
