marcelroed/gigatoken
quality grade B, 67 out of 100Language model tokenization at GB/s
- stars
- 3.8k
- stars gained this week
- +569this week
- forks, open issues and contributors
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Tokenization, parsing, classical NLP pipelines, translation and information extraction.
Signals: nlp, natural-language-processing, tokenizer, named-entity-recognition, text-classification, machine-translation, sentiment-analysis, spacy
633 results
Language model tokenization at GB/s
Student version of Assignment 1 for Stanford CS336 - Language Modeling From Scratch
FinGPT: Open-Source Financial Large Language Models! Revolutionize 🔥 We release the trained model on HuggingFace.
Unsupervised text tokenizer for Neural Network-based text generation.
微舆:人人可用的多Agent舆情分析助手,打破信息茧房,还原舆情原貌,预测未来走向,辅助决策!从0实现,不依赖任何框架。
A multilingual morphological analysis library.
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
✅(已完结)超级全面的 深度学习 笔记【土堆 Pytorch】【李沐 动手学深度学习】【吴恩达 深度学习】【大飞 大模型Agent】
A unified framework for machine learning with time series
Thai natural language processing in Python
Generalist and Lightweight Model for Named Entity Recognition (Extract any entity types from texts)
Taranis AI is an advanced Open-Source Intelligence (OSINT) tool, leveraging Artificial Intelligence to revolutionize information gathering and situational analysis.
Apache OpenNLP
modest natural-language processing
Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages
Agentic AI platform that harnesses Visual LLM Chaining to build proactive digital assistants
This package features data-science related tasks for developing new recognizers for Presidio. It is used for the evaluation of the entire system, as well as for evaluating specific PII recognizers or PII detection models.
An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.
Statistical Machine Intelligence & Learning Engine
scikit-learn: machine learning in Python
No description
textlint is the pluggable linter for natural language text.
NLTK Source
State of the Art Natural Language Processing
24,520 repositories in the index in total.