Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A Japanese tokenizer based on recurrent neural networks
| Date | Stars |
|---|---|
| 2026-07-24 | 418 |
| 2026-07-25 | 418 |
| 2026-07-28 | 418 |
| 2026-07-30 | 418 |
| 2026-07-31 | 418 |
| 2026-08-06 | 418 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<p align="center"><img width="50%" src="/nagisa/data/nagisa_logo.png" /></p>
---
[](https://github.com/taishi-i/nagisa/actions/workflows/python-package.yml)
[](https://coveralls.io/github/taishi-i/nagisa?branch=master)
[](https://nagisa.readthedocs.io/en/latest/?badge=latest)
[](https://pypi.python.org/pypi/nagisa)
[](https://huggingface.co/spaces/taishi-i/nagisa-demo)
[](https://pepy.tech/project/nagisa)
Nagisa is a python module for Japanese word segmentation/POS-tagging.
It is designed to be a simple and easy-to-use tool.
This tool has the following features.
- Based on recurrent neural networks.
- The word segmentation model uses character- and word-level features [[池田+]](http://www.anlp.jp/proceedings/annual_meeting/2017/pdf_dir/B6-2.pdf).
- The POS-tagging model uses tag dictionary information [[Inoue+]](http://www.aclweb.org/anthology/K17-1042).
For more details refer to the following links.
- The documentation is available [here](https://nagisa.readthedocs.io/en/latest/?badge=latest).
- The article in Japanese is available [here](https://qiita.com/taishi-i/items/5b9275a606b392f7f58e).
- The presentation slide at PyCon JP (2022) is available [here](https://speakerdeck.com/taishii/pycon-jp-2022).
Installation
=============
You can install nagisa using pip:
```bash
pip install nagisa
````
Supported Platforms:
- 🐧 Linux: Python 3.6 - 3.14
- 🍎 macOS: Python 3.9 - 3.14
- 🪟 Windows: Python 3.9 - 3.14
Basic usage
=============
Sample of word segmentation and POS-tagging for Japanese.
The output tokens are normalized using Unicode NFKC normalization.
```python
import nagisa
text = 'Pythonで簡単に使えるツールです'
words = nagisa.tagging(text)
print(words)
#=> Python/名詞 で/助詞 簡単/形状詞 に/助動詞 使える/動詞 ツール/名詞 です/助動詞
# Get a list of words
print(words.words)
#=> ['Python', 'で', '簡単', 'に', '使える', 'ツール', 'です']
# Get a list of POS-tags
print(words.postags)
#=> ['名詞', '助詞', '形状詞', '助動詞', '動詞', '名詞', '助動詞']
```
Post-processing functions
=====
Filter and extarct words by the specific POS tags.
```python
import nagisa
# Filter the words of the specific POS tags.
words = nagisa.filter(text, filter_postags=['助詞', '助動詞'])
print(words)
#=> Python/名詞 簡単/形状詞 使える/動詞 ツール/名詞
# Extarct only nouns.
words = nagisa.extract(text, extract_postags=['名詞'])
print(words)
#=> Python/名詞 ツール/名詞
# This is a list of available POS-tags in nagisa.
print(nagisa.tagger.postags)
#=> ['補助記号', '名詞', ... , 'URL']
```
Add the user dictionary in easy way.
```python
import nagisa
# default
text = "3月に見た「3月のライオン」"
print(nagisa.tagging(text))
#=> 3/名詞 月/名詞 に/助詞 見/動詞 た/助動詞 「/補助記号 3/名詞 月/名詞 の/助詞 ライオン/名詞 」/補助記号
# If a word ("3月のライオン") is included in the single_word_list, it is recognized as a single word.
new_tagger = nagisa.Tagger(single_word_list=['3月のライオン'])
print(new_tagger.tagging(text))
#=> 3/名詞 月/名詞 に/助詞 見/動詞 た/助動詞 「/補助記号 3月のライオン/名詞 」/補助記号
```
Nagisa provides a built-in Japanese stopwords list.
```python
import nagisa
# default
text = "日本語のストップワードを簡単に利用できます。"
tokens = nagisa.tagging(text)
print(tokens.words)
#=> ['日本', '語', 'の', 'ストップ', 'ワード', 'を', '簡単', 'に', '利用', 'でき', 'ます', '。']
# Filter out stopwords from the tokenized result
words = [word for word in tokens.words if word not in nagisa.stopwords]
print(words)
#=> ['日本', '語', 'ストップ', 'ワード', '簡単', '利用', '。']
```
Train a model
======
Nagisa provides a simple train method
for a joint word segmentation and sequence labeling (e.g, POS-tagging, NER) model.
The format of Excerpt of 4,962 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:5c303bc7f5f3b6e2, topic:nlp, topic:natural-language-processing, topic:tokenizer