Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A Thai word tokenization library using Deep Neural Network
| Date | Stars |
|---|---|
| 2026-07-31 | 429 |
| 2026-08-07 | 429 |
| 2026-08-08 | 429 |
| 2026-08-09 | 428 |
| 2026-08-16 | 428 |
| 2026-08-17 | 428 |
| 2026-08-18 | 428 |
| 2026-10-27 | 427 |
| 2026-11-04 | 426 |
Today
-1 stars today
This week
-1 stars this week
This month
-2 stars this month
Momentum
0.0
growth rate 0.00%/day
# Deepcut
[](https://github.com/rkcosmos/deepcut/blob/master/LICENSE) [](https://zenodo.org/badge/latestdoi/95091660)
A Thai word tokenization library using Deep Neural Network.
## What's new
* `v0.7.0` Migrate from keras to TensorFlow 2.0
* `v0.6.0` Allow excluding stop words and custom dictionary, updated weight with semi-supervised learning
* `v0.5.2` Better pretrained weight matrix
* `v0.5.1` Faster tokenization by code refactorization
* `examples` folder provide starter script for Thai text classification problem
* `DeepcutJS`, you can try tokenizing Thai text on web browser [here](https://rkcosmos.github.io/deepcut/)
## Performance
The Convolutional Neural network is trained from 90 % of NECTEC's BEST corpus (consists of 4 sections, article, news, novel and encyclopedia) and test on the rest 10 %. It is a binary classification model trying to predict whether a character is the beginning of word or not. The results calculated from only 'true' class are as follow
| Precision | Recall | F1 |
| --------- | ------ | ------ |
| 97.8% | 98.5% | 98.1% |
## Installation
Install using `pip` for stable release (tensorflow version2.0),
``` bash
pip install deepcut
```
For latest development release (recommended),
``` bash
pip install git+git://github.com/rkcosmos/deepcut.git
```
If you want to use tensorflow version 1.x and standalone keras, you will need
``` bash
pip install deepcut==0.6.1
```
### Docker
First, install and run [`docker`](https://www.docker.com/get-started) on your machine. Then, you can build and run `deepcut` as follows
``` bash
docker build -t deepcut:dev . # build docker image
docker run --rm -it deepcut:dev # run docker, -it flag makes it interactive, --rm for clean up the container and remove file system
```
This will open a shell for us to play with `deepcut`.
## Usage
``` python
import deepcut
deepcut.tokenize('ตัดคำได้ดีมาก')
```
Output will be in list format
``` bash
['ตัดคำ','ได้','ดี','มาก']
```
### Bag-of-word transformation
We implemented a tokenizer which works similar to [`CountVectorizer`](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html) from `scikit-learn` . Here is an example usage:
``` python
from deepcut import DeepcutTokenizer
tokenizer = DeepcutTokenizer(ngram_range=(1,1),
max_df=1.0, min_df=0.0)
X = tokenizer.fit_tranform(['ฉันบินได้', 'ฉันกินข้าว', 'ฉันอยากบิน']) # 3 x 6 CSR sparse matrix
print(tokenizer.vocabulary_) # {'บิน': 0, 'ได้': 1, 'ฉัน': 2, 'อยาก': 3, 'ข้าว': 4, 'กิน': 5}, column index of sparse matrix
X_test = tokenizer.transform(['ฉันกิน', 'ฉันไม่อยากบิน']) # use built tokenizer vobalurary to transform new text
print(X_test.shape) # 2 x 6 CSR sparse matrix
tokenizer.save_model('tokenizer.pickle') # save the tokenizer to use later
```
You can load the saved tokenizer to use later
``` python
tokenizer = deepcut.load_model('tokenizer.pickle')
X_sample = tokenizer.transform(['ฉันกิน', 'ฉันไม่อยากบิน'])
print(X_sample.shape) # getting the same 2 x 6 CSR sparse matrix as X_test
```
### Custom Dictionary
User can add custom dictionary by adding path to `.txt` file with one word per line like the following.
``` bash
ขี้เกียจ
โรงเรียน
ดีมาก
```
The file can be placed as an `custom_dict` argument in `tokenize` function e.g.
``` python
deepcut.tokenize('ตัดคำได้ดีมาก', custom_dict='/path/to/custom_dict.txt')
deepcut.tokenize('ตัดคำได้ดีมาก', custom_dict=['ดีมาก']) # alternatively, you can provide a list of custom dictionary
```
## Notes
Some texts might not be segmented as we would expected (e.g.'โรงเรียน' -> ['โรง', 'เรียน']), this is because of
* BEST corpus (training data) tokenizes word this way (They use 'Compound words' as aExcerpt of 5,577 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:0b37ff7aadd50c6e, topic:deep-learning, topic:tensorflow