Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
The most accurate natural language detection library for Python, suitable for short text and mixed-language text
| Date | Stars |
|---|---|
| 2026-07-24 | 1766 |
| 2026-07-25 | 1766 |
| 2026-07-28 | 1766 |
| 2026-07-30 | 1766 |
| 2026-08-06 | 1766 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
5.0
growth rate 0.00%/day
<div align="center">  [](https://github.com/pemistahl/lingua-rs/actions/workflows/python-build.yml) [](https://codecov.io/gh/pemistahl/lingua-rs) [](#4-which-languages-are-supported)  [](https://pypi.org/project/lingua-language-detector) [](https://www.apache.org/licenses/LICENSE-2.0) </div> <br> ## 1. What does this library do? Its task is simple: It tells you which language some text is written in. This is very useful as a preprocessing step for linguistic data in natural language processing applications such as text classification and spell checking. Other use cases, for instance, might include routing e-mails to the right geographically located customer service department, based on the e-mails' languages. ## 2. Why does this library exist? Language detection is often done as part of large machine learning frameworks or natural language processing applications. In cases where you don't need the full-fledged functionality of those systems or don't want to learn the ropes of those, a small flexible library comes in handy. Python is widely used in natural language processing, so there are a couple of comprehensive open source libraries for this task, such as Google's [*CLD 2*](https://github.com/CLD2Owners/cld2) and [*CLD 3*](https://github.com/google/cld3), [*Langid*](https://github.com/saffsd/langid.py), [*Simplemma*](https://github.com/adbar/simplemma) and [*Langdetect*](https://github.com/Mimino666/langdetect). Unfortunately, except for the last one they have two major drawbacks: 1. Detection only works with quite lengthy text fragments. For very short text snippets such as Twitter messages, they do not provide adequate results. 2. The more languages take part in the decision process, the less accurate are the detection results. *Lingua* aims at eliminating these problems. She nearly does not need any configuration and yields pretty accurate results on both long and short text, even on single words and phrases. She draws on both rule-based and statistical Naive Bayes methods but does not use neural networks or any dictionaries of words. She does not need a connection to any external API or service either. Once the library has been downloaded, it can be used completely offline. ## 3. A short history of this library This library started as a pure Python implementation. Python's quick prototyping capabilities made an important contribution to its improvements. Unfortunately, there was always a tradeoff between performance and memory consumption. At first, *Lingua's* language models were stored in dictionaries during runtime. This led to quick performance at the cost of large memory consumption (more than 3 GB). Because of that, the language models were then stored in NumPy arrays instead of dictionaries. Memory consumption reduced to approximately 800 MB but CPU performance dropped significantly. Both approaches were not satisfying. Starting from version 2.0.0, the pure Python implementation was replaced with compiled Python bindings to the native [Rust implementation](https://github.com/pemistahl/lingua-rs) of *Lingua*. This decision has led to both quick performance and a small memory footprint. The pure Python implementation is still available in a [separate branch](https://github.com/pemistahl/lingua-py/tree/pure-python-impl) in this repository and will be kept up-to-date in subsequent 1.* releases. There are environments that do not support native Pyth
Excerpt of 28,211 characters
Read on GitHub
135
65
1
Jiayu Liu · Singapore
1
Guspan Tanadi
1
Marco Castelluccio · @mozilla · Italy
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:8ae14c5771039e93, topic:nlp, topic:natural-language-processing, readme:natural language processing