Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
ETL, document parsing, web crawling, synthetic data generation, annotation and dataset curation tooling.
Signals: data-engineering, etl, data-pipeline, web-scraping, crawler, data-labeling, annotation-tool, synthetic-data
464 results
Use LLMs to robustly extract web data
LLM-powered lossless compression tool
A package for parsing PDFs and analyzing their content using LLMs.
Filter sensitive information from free text before sending it to external services or APIs, such as chatbots and LLMs.
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
clean & curate your data with LLMs.
Parse PDFs into markdown using Vision LLMs
smart-llm-loader is a lightweight yet powerful Python package that transforms any document into LLM-ready chunks. Spend less time on preprocessing headaches and more time building what matters. From RAG systems to chatbots to document Q&A, SmartLLMLoader handles the heavy lifting so you can focus on creating exceptional AI applications.
A live reading list for LLM data synthesis (Updated to July, 2025).
DSIR large-scale data selection framework for language model training
DataMate is an enterprise-level data processing platform designed for model fine-tuning and RAG retrieval.
Local-first, open-source AI assistant for your data. Unify tasks, notes, docs, photos, and bookmarks. Private, self-hosted, and extensible via APIs.
extract all your personal data history from cursor, codex, claude-code, windsurf, and trae
Turn an entire GitHub Repo into a single organized .txt file to use with LLM's (ChatGPT, Claude, Gemini, etc)
🔍 Better text detection by combining multiple OCR engines (EasyOCR, Tesseract, and Pororo) with 🧠 LLM.
DOM to Semantic-Markdown for use with LLMs
PDF to markdown using vision LLMs — tables, layouts, and structure preserved
Twitter data scraping, embedding based image search and more.
No description
This is a public repository to go over all the LLM-driven data engineering concepts.
A fast tool to convert any website into LLM-ready markdown data. Built by https://supermemory.ai
Improved file parsing for LLM’s
A reading list on LLM based Synthetic Data Generation 🔥
[NeurIPS2023] DatasetDM:Synthesizing Data with Perception Annotations Using Diffusion Models
24,545 repositories in the index in total.
