Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
| Date | Stars |
|---|---|
| 2026-07-24 | 296 |
| 2026-07-25 | 297 |
| 2026-07-28 | 297 |
| 2026-07-30 | 297 |
| 2026-08-06 | 297 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# 🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
Vaporetto is a fast and lightweight pointwise prediction-based tokenizer.
This repository includes both a Rust crate that provides APIs for Vaporetto and CLI frontends.
[](https://crates.io/crates/vaporetto)
[](https://docs.rs/vaporetto)
[](https://github.com/daac-tools/vaporetto/actions)
[](https://join.slack.com/t/daac-tools/shared_invite/zt-1pwwqbcz4-KxL95Nam9VinpPlzUpEGyA)
[日本語のドキュメント](README-ja.md)
[Wasm Demo](https://vaporetto-demo.pages.dev/) (takes a little time to load the model.)
A Python wrapper is also available [here](https://github.com/daac-tools/python-vaporetto).
## Example Usage
### Try Word Segmentation
This software is implemented in Rust. Install `rustc` and `cargo` following [the documentation](https://www.rust-lang.org/tools/install) beforehand.
Vaporetto provides three ways to generate tokenization models:
#### Download Distribution Model
The first is the simplest way, which is to download a model we have trained.
Models are available [here](https://github.com/daac-tools/vaporetto-models/releases).
We chose `bccwj-suw+unidic_pos+pron`:
```
% wget https://github.com/daac-tools/vaporetto-models/releases/download/v0.5.0/bccwj-suw+unidic_pos+pron.tar.xz
```
Each file is a compressed file containing a model file and license terms, so you need to decompress the downloaded file as shown in the following command:
```
% tar xf ./bccwj-suw+unidic_pos+pron.tar.xz
```
To perform tokenization, run the following command:
```
% echo 'ヴェネツィアはイタリアにあります。' | cargo run --release -p predict -- --model path/to/bccwj-suw+unidic_pos+pron.model.zst
```
The following will be output:
```
ヴェネツィア は イタリア に あり ます 。
```
##### Notes for Vaporetto APIs
The distribution models are compressed in the zstd format.
If you want to load these compressed models with the *vaporetto* API,
you must decompress them outside of the API.
```rust
// Requires zstd crate or ruzstd crate
let reader = zstd::Decoder::new(File::open("path/to/model.zst")?)?;
let model = Model::read(reader)?;
```
You can also decompress the file using the *unzstd* command, which is bundled with modern Linux
distributions.
#### Convert KyTea's Model
The second is also a simple way, which is to convert a model trained by KyTea.
First of all, download the model of your choice from the [KyTea Models](http://www.phontron.com/kytea/model.html) page.
We chose `jp-0.4.7-5.mod.gz`:
```
% wget http://www.phontron.com/kytea/download/model/jp-0.4.7-5.mod.gz
```
Each file is a compressed file, so you need to decompress the downloaded model file as shown in the following command:
```
% gunzip ./jp-0.4.7-5.mod.gz
```
To convert a KyTea model into a Vaporetto model, run the following command in the Vaporetto root directory.
```
% cargo run --release -p convert_kytea_model -- --model-in path/to/jp-0.4.7-5.mod --model-out path/to/jp-0.4.7-5-tokenize.model.zst
```
Now you can perform tokenization. Run the following command:
```
% echo 'ヴェネツィアはイタリアにあります。' | cargo run --release -p predict -- --model path/to/jp-0.4.7-5-tokenize.model.zst
```
The following will be output:
```
ヴェネツィア は イタリア に あ り ま す 。
```
#### Train Your Model
The third way, which is mainly for researchers, is to prepare a training corpus and train your tokenization models.
Vaporetto can train from two types of corpora: fully annotated corpora and partially annotated corpora.
Fully annotated corpora are corpora in which all character boundaries are annotated with either token boundaries or internal positions of tokens.
This is the data in the form of spaces inserted into the boundaries of the tokens, as shown below:
```
ヴェネツィア は イタリア に あり ます 。
火星 猫 の 生態 の 調査 結果
```
BeExcerpt of 10,573 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:d21d37085f1e5aad, topic:nlp, topic:tokenizer, desc:tokenizer