Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A multilingual command line sentence tokenizer in Golang
| Date | Stars |
|---|---|
| 2026-07-24 | 473 |
| 2026-07-25 | 473 |
| 2026-07-28 | 473 |
| 2026-07-30 | 473 |
| 2026-08-06 | 473 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
[](https://github.com/neurosnap/sentences/actions/workflows/release.yml)
[](https://godoc.org/github.com/neurosnap/sentences)
[](https://goreportcard.com/report/github.com/neurosnap/sentences)
# Sentences - A command line sentence tokenizer
This command line utility will convert a blob of text into a list of sentences.
* [Demo](https://sentences-231000.appspot.com/)
* [Docs](https://godoc.org/github.com/neurosnap/sentences)
## Features
* Supports multiple languages (english, czech, dutch, estonian, finnish,
german, greek, italian, norwegian, polish, portuguese, slovene, and turkish)
* Zero dependencies
* Extendable
* Fast
## Install
### arch
[aur](https://aur.archlinux.org/packages/sentences-bin)
### mac
```
brew tap neurosnap/sentences
brew install sentences
```
### other
Or you can find the pre-built binaries on [the github
releases page](https://github.com/neurosnap/sentences/releases).
### using golang
```
go get github.com/neurosnap/sentences
go install github.com/neurosnap/sentences/cmd/sentences
```
## Command
## Get it
```
go get github.com/neurosnap/sentences
```
## Use it
```Go
import (
"fmt"
"os"
"github.com/neurosnap/sentences"
)
func main() {
text := `A perennial also-ran, Stallings won his seat when longtime lawmaker David Holmes
died 11 days after the filing deadline. Suddenly, Stallings was a shoo-in, not
the long shot. In short order, the Legislature attempted to pass a law allowing
former U.S. Rep. Carolyn Cheeks Kilpatrick to file; Stallings challenged the
law in court and won. Kilpatrick mounted a write-in campaign, but Stallings won.`
// download the training data from this repo (./data) and save it somewhere
b, _ := os.ReadFile("./path/to/english.json")
// load the training data
training, _ := sentences.LoadTraining(b)
// create the default sentence tokenizer
tokenizer := sentences.NewSentenceTokenizer(training)
sentences := tokenizer.Tokenize(text)
for _, s := range sentences {
fmt.Println(s.Text)
}
}
```
## English
This package attempts to fix some problems I noticed for english.
```Go
import (
"fmt"
"github.com/neurosnap/sentences/english"
)
func main() {
text := "Hi there. Does this really work?"
tokenizer, err := english.NewSentenceTokenizer(nil)
if err != nil {
panic(err)
}
sentences := tokenizer.Tokenize(text)
for _, s := range sentences {
fmt.Println(s.Text)
}
}
```
## Contributing
I need help maintaining this library. If you are interested in contributing
to this library then please start by looking at the [golden-rules](https://github.com/neurosnap/sentences/tree/golden-rule) branch which
tests the [Golden Rules](https://github.com/diasks2/pragmatic_segmenter/blob/master/README.md#the-golden-rules)
for english sentence tokenization created by the [Pragmatic Segmenter](https://github.com/diasks2/pragmatic_segmenter)
library.
Create an issue for a particular failing test and submit an issue/PR.
I'm happy to help anyone willing to contribute.
## Customize
`sentences` was built around composability, most major components of this package
can be extended.
Eager to make ad-hoc changes but don't know how to start?
Have a look at `github.com/neurosnap/sentences/english` for a solid example.
## Notice
I have not tested this tokenizer in any other language besides English. By default
the command line utility loads english. I welcome anyone willing to test the
other languages to submit updates as needed.
A primary goal for this package is to be multilingual so I'm willing to help in
any way possible.
This lExcerpt of 5,418 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:478fece40a586953, topic:tokenizer, desc:tokenizer, readme:tokenizer